Drew Black broken crankshaft
 8009 Posts 9/99
| Posted - Jan 27 2020 : 10:37PMI'm having some problems with the software that syncs images from the server that handles the uploaded files to the 2 primary servers that generate the forum pages. I'm manually running a re-sync now from the master server to the 2 slaves. This should fix it. The sync could take a few hours. There's about 200GB of data. I've been working on generating alternate versions of images for browsers that specify that they want to save data. And I'm generating 100 x 100 thumbnails for all 475k images on the server to support the site redesign that's in-process. [Link] |
Mass Facialties Deactivated User
"Facialized Vol. 7" 2021 AVN Awards Winner: Best Oral Movie 9308 Posts 9/13
| Posted - Jan 27 2020 : 10:51PM
^ Thanks for the notification. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jan 27 2020 : 10:59PM
There's a little over 2.8 million files that need to be synced up. Each image that's uploaded can end up being processed into to 6 to 10 different images depending on the height x width of the original. This will take a while. I'm guessing 7 or 8 hours.
 [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jan 28 2020 : 7:24AM
The full re-sync has been running almost 9 hours and is 46 percent complete at the moment.added: 11.5 hours into it and the process is at 76%. Edited by - Drew Black on 1/28/2020 8:14:49 AM Edited by - Drew Black on 1/28/2020 9:52:51 AM  [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jan 28 2020 : 7:12PM
This problem should be fixed now. [Link] |
| |
|
Mass Facialties Deactivated User
"Facialized Vol. 7" 2021 AVN Awards Winner: Best Oral Movie 9308 Posts 9/13
| Posted - Jan 28 2020 : 11:36PM |
FlacFan Deactivated User
10088 Posts 8/15
| Posted - Feb 20 2020 : 2:10PM
Hi Drew. It may be acting up again. A pic I posted in the Remy Lacroix ADT Thread didn't post. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Feb 20 2020 : 2:36PM
Yep, I can see it made it to one of the 2 replication servers. I've started a manual replication again to re-sync everything.
 [Link] |
FlacFan Deactivated User
10088 Posts 8/15
| Posted - Feb 20 2020 : 2:46PM
Thank you Drew. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Feb 20 2020 : 2:56PM
It will likely take about an hour to 90 mins. 12% done now.
 [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Feb 21 2020 : 6:03AM
The folder that stores the image uploads is getting too large for the synchronizing process to copy the files quickly. The sync used to take about 8 to 10 seconds top copy a dozen or so files. Not it's taking 1:30 to 2 mins. I added some behind the scenes features recently to support mobile browsers that use the "Data Saver" option. It doubled the number of images on the server.I will get to work on fixing this on Sunday. I'll split up the folders into something more manageable for the synchronizing software. There will be some period where images are broken as I move data around.  [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Feb 22 2020 : 4:32PM
I'm working on a plan to permanently fix this issue. It's going to involve updates to every page and background that deals with uploaded images. It's going to take a few days to implement. I have tested the sync process I plan to use with the new system and it runs in 8 to 10 seconds which should be sufficient. [Link] |
FlacFan Deactivated User
10088 Posts 8/15
| Posted - Feb 22 2020 : 4:33PM
Thank you Drew. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Feb 23 2020 : 9:08AM
I'm testing the program that restructures the storage paths. Trying to get a feel for how long this will take to reshuffle things into smaller bins.
 [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Feb 24 2020 : 8:45AM
The restructuring of user uploads should be complete. I need to do some more testing and monitoring but things appear OK.The old system had everything in 1 "bin" for syncing to the other servers. I've set it up so there are 99 bins each with a subset of user's uploaded files. The syncing should complete within 10 seconds now.
 [Link] |
| |
|
HumptyRumpty Senior Member
1663 Posts 8/18
| Posted - Mar 1 2020 : 12:47PM
Sorry, but it's broken again. The "Favorite Big Natural Breasts" and "Perfect fake tits - pornstars, models and studios" are 2 examples. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Mar 1 2020 : 12:57PM
Could you provide links or more information on what's not working in those 2 topics? I don't see any images that aren't loading in either of them. [Link] |
HumptyRumpty Senior Member
1663 Posts 8/18
| Posted - Mar 1 2020 : 2:18PM
DOH! It's working now. When I reported the problem, all of the places where photos were supposed to be were just an outline of a box, with the standard AdultDVDTalk cream-colored background color inside the box. The boxes also had the little Internet-standard icon that means, "image not available." Don't know the name of the icon. Sorry about that somewhat vague description. Oh well. Glad it's working now. [Link] |
philly26 Senior Member
Fly Eagles Fly Super Bowl LII Champions Nova Nation 2016 and 2018 National Champions 5180 Posts 2/16
| Posted - Mar 3 2020 : 11:21AM
I can't upload images in private conversations as well as in the forums. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Mar 3 2020 : 11:48AM
I'm working on that site at the moment. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Mar 3 2020 : 1:20PM
I was trying to fix some problems that ended up not being related to any of the code that runs this website. The problems are at our CDN provider. Apparently they have been having issues on and off for the past 5 days and didn't send out a notification to their customers. I became aware because I contacted their support group after debugging the mess for about 4 hours this morning. We received a DMCA takedown request earlier today and we try to process them ASAP and reply with proof of removal to the person/agency making the request. Wasn't able to do that today because of the issues at the CDN provider. [Link] |
killbillvol69  All-Star Member
^Lucy Pinder 18631 Posts 4/08
| Posted - Mar 14 2020 : 12:21PM
I've seen several posts made this morning that have pics missing. Maybe the problem is happening again.This post is just one example: Help with ID of this Spanish Star edit: I came across a couple of people complaining that they couldn't upload pics as well, and I thought they meant that they didn't know how, but probably they meant it wasn't working. Like this post. Edited by - killbillvol69 on 3/14/2020 12:26:34 PM  [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Mar 14 2020 : 1:09PM
Yes. It is happening again. All kinds of problems on the server the past few days since the most recent batch of windows updates made a mess of things. The sync service is crashing constantly and our backup software is mo longer functional too. Rebooting the primary server now and hoping i can figure out WTF is wrong [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Mar 14 2020 : 6:18PM
I have reconfigured all 213 bin sync profiles on the primary server to notify me via email of any failures. As long as I'm near my phone or computer I should be aware of problems fairly quickly. [Link] |
killbillvol69  All-Star Member
^Lucy Pinder 18631 Posts 4/08
| Posted - Jun 1 2020 : 6:06PM |
| |
|
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jun 1 2020 : 6:31PM
I'm reprocessing the data-saver images and it broke the sync software. I'll fix it in an hour or so when the image processing is done.
 [Link] |
killbillvol69  All-Star Member
^Lucy Pinder 18631 Posts 4/08
| Posted - Jun 1 2020 : 11:19PM
Thanks Drew. [Link] |
Simple Simon Senior Member
12302 Posts 2/15
| Posted - Jun 2 2020 : 12:10AM
I've also noticed that when you upload an image, it often appears in both the upload page and the preview page as a broken image. [Link] |
Simple Simon Senior Member
12302 Posts 2/15
| Posted - Jun 3 2020 : 11:21AM |
Simple Simon Senior Member
12302 Posts 2/15
| Posted - Jun 7 2020 : 7:07AM |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jun 7 2020 : 8:33AM
Pictures are not worth 1000 words. A link to the post would help. If you can't take a moment to provide some type helpful description to help me understand/diagnose/replicate the problem I don't feel a need to go chase this down or pay attention to your posts.As a starting point, email me the original Bon Jovi cover image you uploaded that resulted in a broken image. drew@adultdvdtalk.com You are uploading from your phone? Do you see the image in the upload window after it completes? I'm not able to replicate the problem on my end and I did look into it recently. Edited by - Drew Black on 6/7/2020 8:40:36 AM  [Link] |
Simple Simon Senior Member
12302 Posts 2/15
| Posted - Jun 7 2020 : 9:31AM
^ Hi Drew,A link to the post is here. What's The Latest Album You've Listened To? I've emailed you the original image. Yes, I am uploading from my phone. As per the first screenshot, the image does not appear in the upload page; just a small coloured box appears. Then in the preview screen, as per the second screenshot, just the filename appears. When I go back and forth multiple times, the image often then appears in the preview screen. When I then post the comment, the image usually appears, but sometimes it is just the box/filename. But it always later appears as the image. Just so I use the correct terminology, what do you call it when the image appears as just a small box or filename, as per the screen shots? If you need any further details, just let me know. Thanks in advance.  [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jun 7 2020 : 9:43AM
Thanks for the email. I'll dig into this over the course of the day today. [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jun 7 2020 : 9:54AM
I see the image is working in that thread. Did you have to upload it again to get it to work or did it work the first time?At a quick glance I think the issue is related to the image size. It's a little over 2.5MB and >2500px wide. There's a memory limit on the upload page script to prevent a denial-of-service situation where people could upload huge images and overwhelm the server's resources. This image might be bumping up against that limit. The image that appears post-upload in the uploader screen and the post preview window comes from the https://upimg.adultdvdtalk.com site. This is a sandbox site that doesn't share any resources with other parts of ADT. After the image is processed on our end it gets moved to it's final location and then replicated to the mini web farm of 3 servers that process requests for ADT.
You're getting broken images from the sandboxed site it tells me something failed in the upload process. Slow upload/processing timeouts, too much memory, image too big, etc. You might be sending a large file half-way around the world and something is getting tripped up in the process.  [Link] |
Simple Simon Senior Member
12302 Posts 2/15
| Posted - Jun 7 2020 : 7:51PM
^ For the image in question, I didn't load it again. I got the 'broken image' in both the upload and preview screens, but then the image appeared when I loaded the post. This is usually what happens, so I dont think it's an upload issue.Yesterday I went in and out of the preview screen, and on about the third or fourth time it changed from a 'broken image' to the actual image. Also, I do sometimes get the 'broken image' when I post the comment; and then I later go back to the post and the image appears. In the screen shots I posted, can I ask if you've seen those boxes before? Look, it's not a huge issue - it just means I generally can't see images in the preview screen. But just noting, I do sometimes have this issue in the live posts.  [Link] |
| |
|
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jun 8 2020 : 8:18AM
There's a few issues at play here. When you're uploading an image you're connecting directly to our server located in Brunswick, Maine, USA. I think that might be a long way away from where you're located. I'm not sure why the images are appearing broken in the upload window and preview window. I'll need to look at a few server configuration options and look at some log files to diagnose what's going on there.After an image is processed into alternate sizes for mobile browsers it is synced to 3 other physical servers. The processing and syncing usually takes 3 - 5 seconds but if you have uploaded a lot of images over the years the sync process could take up to 30 seconds. The sync delay should not not have any impact on the upload window or preview window. That's what's confusing me. Those two pages use the image that was just uploaded in its pre-processed form. (except for height x width processing/shrinking. HxW reductions happen in the uploader process.) The HTML that generates a forum page comes from a server in Maine, USA. The images that appear in live topics are served from our content delivery network provider. Their servers are located all over the world in an effort to provide more localized delivery. If they don't have a copy of the image on their servers they need to fetch it from our source server in Maine. This source-fetch-deliver process slows things down for the first request. Since you just uploaded the image you're likely the first person to request the image too.  [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Jun 8 2020 : 9:53AM
This should be fixed now. The problem was related to the way we handle the "Save-Data" header that's sent by some phones. The data saver image versions are not processed immediately upon upload. They are created in a subsequent processing step. I've added some code to check if the save-data version is available and if it's not it will send the full-sized image.
 [Link] |
Simple Simon Senior Member
12302 Posts 2/15
| Posted - Jun 8 2020 : 5:09PM
^ I just tested it, and the image appeared in both the upload and preview screens. Thanks Drew. [Link] |
Mass Facialties Deactivated User
"Facialized Vol. 7" 2021 AVN Awards Winner: Best Oral Movie 9308 Posts 9/13
| Posted - Sep 20 2020 : 12:08AM |
Brummie Senior Member
3389 Posts 1/09
| Posted - Sep 20 2020 : 6:16AM
^ Hi MassDrew's fixed it now. Your image should be visible. Mine are.  [Link] |
Drew Black broken crankshaft
8009 Posts 9/99
| Posted - Sep 20 2020 : 6:26AM
The new WAF software blocked some of the IP addresses of our CDN provider. [Link] |