Server issues 9/28/15

ZenBrianZenBrian Administrator Posts: 1,689
We have had several site outages in the past week linked to one of our database servers. We took the site down on Saturday to upgrade the server.

The site was fully functional after that, but today, we started experiencing some database server instabilities. The team is currently working on assessing and fixing the issue but we don’t have the final assessment of the situation yet.

We know this is impacting you and your business and we sincerely apologize. We will let you know as soon as we know more.
Thanks!

Brian
Zenfolio Support
image


Check out my site! www.bussierephoto.com

Comments

  • NOT GOOD ENOUGH!

    Maybe it's time to send your monkeys back to the zoo and employ some competent technicians for once.

    There is no excuse for the sort of rubbish level unreliability that Zenfolio has become. Blaming the technology doesn't cut it. Its the monkeys you are employing to run it that are the problem.
  • rickryanrickryan Member Posts: 1
    Wow. Everybody said "zenfolio" was the best platform. Guess I should've gone with smugmug. I'm a tech guy and when you have a server go down, throwing new software changes isn't the answer. You stabilize the platform back to where you had it before the problems started.
  • cphotocphoto San FranciscoMember Posts: 81
    Zenfolio should be up 99.99% of the time with a proper architecture, redundancy, etc. Having the service down for HOURS several times a week is totally unacceptable and unprofessional.
  • ZenBrianZenBrian Administrator Posts: 1,689
    The site is back up and running smoothly! Thank you all so much for your patience today. We will follow up with everyone directly with more information about today's problems and findings.
    Thanks!

    Brian
    Zenfolio Support
    image


    Check out my site! www.bussierephoto.com
  • I dispair.
    The forum is full of spam and the web sites are running at a snail's pace.
  • Site will not load, "contact us" and "support center" won't load.
    In fact the only part of Zenfolio that does work is this forum.
  • 1stclassimages1stclassimages ukMember Posts: 3
    ZenBrian said:

    The site is back up and running smoothly! Thank you all so much for your patience today. We will follow up with everyone directly with more information about today's problems and findings.

    Well thats not true
  • andreasweberandreasweber Stuttgart, GermanyMember Posts: 1,001
    With the 21 minutes unscheduled downtime earlier today it's now 16 hours 14 minutes down in the last 14 days - more than one hour per day (pretty exactly 5% downtime)!
  • "The service is unavailable."

    21.10 GMT

    Farcical
  • The farce has gone on too long.

    We received a seemingly genuine letter from Mr Collin stating that the problem had been fixed. Right now that seems like a pack of lies. Is the promise, in that same letter, of moving the system to a proper hosting platform with redundancy to there will be no more down time even credible?

    It isn't fixed at all. Ok, to be specific it was fixed for about two hours, but now its right back to being the same variable between so slow it unusable to just plain not accessible.

    Is Mr Collin now also lying to us, or is he just parroting the lies fed to him by his IT staff? Either way Zenfolio looks very very bad and will have no credibility until the lies end and the problems are actually fixed.

    The big question is should we even continue to waste out time, loose business, and put up with our customers being hosed off because the best Zenfolio can achieve seems to be excuses and hollow promises.
  • neuPhotosneuPhotos Member Posts: 2
    I now have customers complaining they can't order their high school son's football sports action photographs, parents wanting images for senior yearbook pages due in the next two weeks, and a mom who wanted to review proofs of her daughter's senior Images from this weekend.

    Zenfolio - You are making us look bad to our customers. ZenBrian or Zen"whoever" need to be providing more frequent updates so I can update my customers as they are complaining. Also, this is now at a point where you need to extend a compensatory offer to all your customers. We await your compensation offer and a TRUE & a no PR assessment of what has occurred and how it will NOT happen again.
  • cphotocphoto San FranciscoMember Posts: 81
    This is ridiculous.... Was trying to show my site to a customer 30 minutes ago, and do what do I get?
    We are currently performing system maintenance. We apologize for any inconvenience and appreciate your patience while we fix the issue.

    Come on guys!!! We can't wait until next year to have a fully redundant and properly architectured system.

    Right now you make us look like amateurs.
  • andreasweberandreasweber Stuttgart, GermanyMember Posts: 1,001
    To be fair - we could have known a long time: For years now every tiny feature upgrade and any maintenance required hours of downtime of the productive system. Typically those were announced a mere three days in advance (which can hardly count as "scheduled", since any customer appointments would have been fixed far earlier ...) And recently the planned downtime typically was followed by several hours more unscheduled downtime to get the system working and some weeks of bug fixing on the live system. So there never was any redundancy, testing before deployment, or appreciation of customer needs - IOW, an alarming lack of professionalism at the core of performance.
    Any serious system failure just had to lead to desaster ...
  • andreasweberandreasweber Stuttgart, GermanyMember Posts: 1,001
    Update: by now the last 7 days alone saw 17 hours and 46 minutes downtime - that's less than 90% uptime in that period!
  • cphotocphoto San FranciscoMember Posts: 81
    edited October 2015
    Site is slowly dying again, it's taking minutes to load my customer galleries. I'm guessing we'll get the "maintenance screen" pretty soon to reboot the DB server... Someone just tell me it's only a bad dream?
  • neuPhotosneuPhotos Member Posts: 2
    Sorry cphoto it's not a dream...it's a nightmare!

    This is so unacceptable. It was working great this morning and now an "Error 500: Internal Server" error. My workflow just doesn't include downtime due to server errors and my clients doesn't either. Now I have to upload images to another location so a customer can download them for a project they need to work on tomorrow or deliver them on a CD. ZenMaster - it's time to get the parent company (Art.com inc) involved and spend some serious money on getting this fixed; that means outside experts to parachute in and save our livelihoods. My name is on the website not Zenfolio and it's the one being tarnished because of you.
  • cphotocphoto San FranciscoMember Posts: 81
    Well then it's time for plan B, I can't no longer tolerate that they hurt my reputation and sales. Time to find another provider. Did anyone switch to smugmug and happy with the services? If they are reading this forum a good marketing move would be to offer a discount / migration help for customers willing to switch.
  • cphotocphoto San FranciscoMember Posts: 81
    Looks like the site is up again. It was down for "just" about 2 hours so far today! From twitter: "All site functionality has been restored, and in-depth analysis verified/complete! Thanks so much for hanging in there." I think read that earlier this week. Should we expect the site to be down for 3 hours tomorrow?
  • Kevin KrowsKevin Krows United StatesMember Posts: 1,451
    edited October 2015
    Hard to stay out or away from this conversation. A few more thoughts: (Mr. Collin)

    Believe me, there's nobody more frustrated and disappointed about the lack of service (slow to no service and uploading issues) since the third week of May 2015 when all of these issues started popping up. As I stated previously, this is totally unacceptable in a "enterprise" environment as systems should be monitored 24/7 365 days a year with contingency plans, redundancy, procedures, and protocols to insure 100% up time ... all the time. Let's be honest, nobody cares or remembers the 99.9% when the systems are up and running. What everyone does care about is the .1% when they aren't up and running.

    Anything less than 100% is nothing the promote or brag about. In fact, 100% up time is no longer the exception .... IT'S THE RULE!! I, along with many others here in this forum with IT backgrounds, know first hand that 100% is realist and achievable. Regular system maintenance and upgrades are excluded from the 100% as these events are forecasted and planned in advance. Even my own clients "Get It" and "Understand It" when they see a message that say, "Our system is currently unavailable for maintenance until 8:00 a.m. Central Time. Thank you for your patience. "

    I don't want to take anyone's right to express their frustration provided that it's not a personal attack using name calling etc. Everyone needs to conduct themselves professionally even if you have steam coming out of your ears. Also, making threats that you are looking elsewhere or considering other options is fine. However it won't do anything to speed up the process of solving the real problems. In fact, you may be putting yourself at greater risk for business loss as other services have issues as well. You don't want to risk trading one problem for another. I'm as familiar with other services and talk to other photographers who use them so I know this to be a fact.

    Likewise, Zenfolio support needs to stop apologizing, making "soft landing" comments, or making statements that might not be 100% accurate. Endorsing or supporting that customers may need to leave Zenfolio is also not the answer. What support needs to understand is that often times what they say only makes people more frustrated particularly when the facts are omitted or when they have to eat their words 24 hours later... "Everything is back up and running smoothly." followed by a 4 hour brown out the next day.

    Finally, as I previously stated, this situation falls on the shoulders of one person and one person only. Mr. Collin, General Manager of Zenfolio, needs to be the only voice we need to hear as it's the only one that matters at this stage in the game. Although he has only been with Zenfolio since late May 2015, he has had more than enough time to assess the priorities and determine what changes need to be made. With his background and experience he doesn't need to be reminded that a stable platform is #1 on the list. It's simply the basis from which everything else is made possible from new features to business growth.

    Mr. Collin, if you are reading these messages, I encourage you to email Zenfolio Photographers on a regular basis and give us the hard facts and the overall plan for resolution. We know the difference between a Band-Aid solution vs a permanent fix. We are big boys and big girls so we can handle the truth. You need to do this to restore the integrity of your company as well as establish your own integrity. Don't apologize, don't soft sell, don't compare other services shortcomings, and don't make excuses to justify your situation. If people want to leave Zenfolio then they will at least be doing so based on facts....not frustration. Based on the circumstances, the only voice that matters is yours and it needs to be heard regularly until the systems have stabilized and are at 100%.


    Enough said.
  • mheimannmheimann Member Posts: 1
    Time indeed to switch to another provider. Too bad for all the work performed on zenfolio - was once a very nice site an worth the money. But not anymore.
  • "The site is back up and running smoothly! Thank you all so much for your patience today. We will follow up with everyone directly with more information about today's problems and findings."

    Bullshit.
  • peterjtaylorpeterjtaylor Member Posts: 4
    It's important to note all the points made by 'kevinkrows' who says "there are many of us with backgrounds in IT as well as applications development (and not from a small business perspective either)". If you do the research for this case, it's clear that IT network and systems are unlikely to be the problem here. It's appropriate to blame the poor quality of service on Zenfolio's top management team, who are accountable to the company's stakeholders and customers. The product itself is in jeopardy now, and there should be an all-hands meeting of the top team, product managers and network team to resolve that. In parallel, there should have been an appropriately sophisticated marketing management plan to minimise short-term churn and encourage us all to stay with Zenfolio as our primary provider.
  • ApertureToZenfolio stopped working....is this part of the system failure?????
  • Martin T PhotosMartin T Photos Member Posts: 2
    Hi, what is the ETA to normal service? I had problems just logging in 2 days ago (7 Oct).

    There is not enough information on current status and ETA to normal service. Why the hell don't you have an external service status page yet, it takes 15 min to put a static HTML page on S3 and have Cloudfront serving it.
  • Martin T PhotosMartin T Photos Member Posts: 2
    I just asked support about a refund and I got this.

    "All accounts come with a 30 day money back guarantee. After 30 days, no refunds are provided.

    I’m sorry for the recent frustrations. Even with the recent downtimes, Zenfolio has an excellent uptime record of over 99%. Unfortunately, there were troubles that happened in a short period of time. Our engineers worked hard to get the system back up as quickly as possible."

    Ironically I tried to login just after getting this message about their "excellent uptime record of over 99%" and it took me 3 retries and about 2 minutes to actually login to Zenfolio.

    Bye bye Zenfolio.
  • "We are seeing a delay in image processing after upload. Thanks for your patience while the system works through them all."

    Small type at top of page. Maybe better to just print it as part of the contract when people sign up :)
Sign In or Register to comment.
Help test new Zenfolio features by applying to become a Beta Tester. More info here.
Join our live streaming Zenfolio Customer Support Q & A session on You Tube!

Thursday's at 2pm EST 11am PST

NOTE: You can sign in using your Zenfolio account! To link both accounts, make sure both email addresses match before signing in.
If you have a new feature idea that you think would be a great addition to the Zenfolio service please submit it to our feature voting system.