Migs, the high priest of Linux and server administration, whom I always look to for advice on such things, told me to write about my latest hosting issues and so I shall. I suppose this is a good way to talk and archive the recent problems I’d been having with the sites and what I do to fix it, and how I’m starting to realize that whatever ‘fix’ I do seems temporary since the site’s traffic increase again anyway that as soon as I enhance / improve performance, traffic increases again. I am therefore plagued, within days, by either mysql inefficiencies or run out of RAM. Anyway here’s a rundown:
Here’s a screenshot of Aug. 30’s mysql issues. WordPress serves that up when it can’t connect to the db. I emailed Rimuhosting about it and they figured out I was having issues with one of my favorite WordPress plugins called Counterize, which allowed me to produce those interesting stats at the footer of KikayEx which says ‘x number of visitors visited today, xx number the past 7 days. xxx number total since forever, and so on…‘. We activated an option on mysql to produce a ‘mysql-slow.log‘ which lists down all the mysql queries that take more than 2 seconds to complete. The results were stunning, as Counterize apparently had queries that take as long as 20 seconds, and at a given morning (not even a whole day), there were 40 queries already. Here’s how it looked:
I have a laundry list of things to do on my websites. I do not have the time to do them, and so I am looking for someone with some experience to do them for me.
Sample jobs are: update wordpress, re-activate some domains, update some plugins, check consistency with Internet Explorer, deal with my host re some mod_rewrite issues, and so on. If you, or someone you know, is knowledgable in WordPress, PHP, MySQL, or any combination of the three, please email me at firstname.lastname@example.org . I will prefer that you live in Manila, and can be paid via g-cash.
I will hire you or this person just to execute this list, and for occasional jobs in the future.
From the server grease monkeys at my host:
Hi Gabriel, I logged into your server to investigate the problem and I’ve found that
you’re in dangerous territory concerning memory:
> root@exchange:~# free -m
> total used free shared buffers cached
> Mem: 256 250 5 0 0 1
> -/+ buffers/cache: 249 6
> Swap: 95 95 0
That shows you have 6MB of free RAM, but at some point you’ve exhausted
your entire SWAP Allocation (see last line). If the load had been any
greater, your server would have been killing processes (which would show
in /var/log/messages) in order to remain stable.
So to make the long story short, I had to buy 96mb. more RAM to handle the load. The load usually spikes around lunchtime, exactly when the office girls take a break to check out Kikay. I know that sounds funny, but it’s my best guess and historically, our stats support it. During weekends, hardly anyone views it, with less than ten views per hour. Then on weekdays, especially around noon, 30+ views per hour, viewing as much as 82 pages at the same time.
Warning, highly techie post. All those looking for my usual touchy feely posts please move along now, nothing to see here. Just documenting a disaster of a Permalinks error on WordPress below, which I posted at the WordPress Forum. Posting it below again to document it, to help anyone who has to go through same and so I can keep a record of this sorta crap.
All my 2.1.3 wordpress sites, which had been around since wp 1.5, used to use the structure:
ever since a recent server host move, the sites have started to insist on the ff. structure:
Appending the ‘index.php’ to all my URLs is a complete disaster as I use the Click Counter by Ozh plugin to count my most clicked links. These all now go to 404 pages. I really wanna keep using this as it has been recording my hits since forever.
I tried putting it back to without the index.php but it doesn’t work out, resulting in errors.
John Roberts of OpenDNS was kind enough to drop by to educate my behind on OpenDNS’ ability to refresh their cache, and consequently pointed me to cache.opendns.com, where I found the culprit!
I cleared that out, and apparently only Kikay had that problem. Mom, Basketball, Movie and the rest were ok. After checking them all out, this is the page that says everything’s ok:
I gotta tell ya deleting that record had the same effect as stamping out a cockroach! Or deleting a virus that had a chance to propagate again once accessed.
Day 2 of the big server move is highlighted by three issues.
- Turns out php’s default 8mb allocation was the culprit when I was doing stuff like accessing wp-admin’s pages and it was returning white empty pages. I bumped it up to 16mb. and immediately, that wasn’t enough too. So I bumped it up to 24mb. and I haven’t heard a peep since.
- Around this afternoon all the sites suddenly got MySQL errors of the highest order. Stuff I’ve never seen before, like tables loading but not fetching data (“mysql disappeared” messages) or the default “I cannot connect to the WP db are you sure your password / username is correct” sort of stuff, aside from getting bog slow after initially being lightning fast early on. After a quick support email it turned out the installed 160mb. RAM wasn’t enough to handle the load, and they offered me a 96mb. upgrade for a discounted extra $10.00 a month. I said YeeaAAayyy aLLelujaaaaAhh and they set it forth onto the server posthaste. Looks ok so far although I’ve received some complaints of slowness, so we’ll see.
- Finally the biggest issue of all is DNS. And I mean 3rd party DNS and not my server DNS (thank God). After an hour or so of working on my sites, I try again and get 404 errors. After delving into it I realized I’ve been pointed back to my old server (!?!?!!). Yet another support email, and cause for alarm was raised, and they started asking me to check out stuff. After some settling down we both realized it was probably OpenDNS (which I use on my PC) maintaining an old record which pops up from time to time. Although I’m not particularly sure how and why that is, since I am dependent on that server it suddenly fetches the wrong record after the requisite 10 minute rotation. Now that is just plain weird. At any rate, the only solution is to just wait it out. I initially thought that it was some sort of failback issue, where it serves up a ‘second best’ record should the first one fail for some reason, but the tech support guys said DNS doesn’t work that way so that’s not it. I’m glad (although I was initially freaked) that it turns out the problem is confined to me. Which is freaky still in a way considering I’m it’s admin, and I’m the one having probs accessing it. Figures, right?
Anyway that’s it. I can’t keep my eyes open any longer. Tiring day, and the promise of another again tomorrow. Later then.
wish me luck!
update: as of 17:46 I’ve managed to upload everything, and finally changed the DNS entries to point to my new host. Although the recommended waiting period is 12 hours it should probably be ok before midnight. Meanwhile I’m gonna get a beer and rest a bit. That was a solid 8+ hours of work (off and on since 8am), delayed mostly by the length of time it takes to back up, transfer and activate my gargantuan databases. Crossing fingers.
update: woke up to work on it at 2am and at 4am, and finally 6am. By now, 835am, everything seems to be working normally. There were some issues with too little RAM allocated by php.ini. This did not allow some internal wp-admin pages to load, resulting in a white screen. I upped it to 16 which still wasn’t enough, and now to 24 and is now pending observation. LFTP’s mirroring was also not recursive, and did not migrate the directories write permissions, which is a drag, but not a complete disaster as there are only a few specific dirs that need tweaking. I needed to reload and re-cache my browser several times and reboot, and even removed the OpenDNS IPs which I use, then put them back again, and now finally I can see all the sites. Apparently I should have used rsync, but they (my host) put LFTP there for me so I thought that was the SOP. Anyway..
I’m really tired now. The sites look ok so far so I think I’ll go to sleep, after which I need to finish an article for SME Insight and get it in before lunch. I hope my editor doesn’t read this.
The last command I’ll enter, hopefully for awhile is:
ln -sf /usr/share/zoneinfo/Asia/Manila /etc/localtime, which results in: Mon May 21 08:55:09 PHT 2007. Now to zzzz.
If you entered Pisces-iscariot on your browser, or otherwise clicked a link going to Pisces-iscariot, please don’t adjust your sets. This is now my new blog, which I’ve moved to partly to separate my personal blog from the Exchange.ph domain and partly to practice the eventual migration of the Exchange.ph domain and subsequent websites to a new host.
So I’ve been busy practising a big move to a new server, and this is the first of many migrations I’m about to do in the next few days. Everything looks like it’s working fine now, so I’m pretty happy and hella relieved as well. Below is a shortlist of what I’d had to do to transfer Pisces-Iscariot blogposts, comments and pics lock stock and barrel to this brand spanking new one. This is so that I won’t forget when I have to do it again, with some extra details for whoever else might need the same information. Believe me I’d had to do a lot of groping in the dark (ie. searching in forums, sending tons of email to my host’s excellent support team, testing on my practice server, bugging the chief, etc.) to finally be able to do what I did, so if I’d have appeared extra nervous or cranky these past few days, now you know. At any rate this’ll help anyone who’d happen to be in the same boat I’d be happy. Read on:
Yup. Bored as hell. So I made one while listening to the Beastie Boy’s Intergalactic.
Sigh. As if I didn’t have anything else better to do. hOkay back to work. Merry Christmas everyone!