HomeAboutMailing ListList Chatter /0/0 54.236.59.154

Stupid Sysadmin Mistake of the Minute

2020-02-13 by: "Mike (meuon) Harrison"
From: "Mike (meuon) Harrison" 
------------------------------------------------------


So in an ancient piece of code written a long long time ago, it wrote out 
little tiny log files in a directory...1 per event, timestamp-ish filename.

Absolutely awesome for debugging... but should not have been left on for 
production. After umpteenbazillion files (unknown) have been written, in 
ONE directory.... the journaling file system gets really bonkers.

Yes, by spec it is "unlimited" with some caveats about available inodes, 
etc. but in practice: Don't go there.

That code doesn't do that anymore.. no need, it's been stable (except 
for this...) for a very long time.

I'm sharing for catharsis, and so that you if notice your IO getting 
really slow (check iotop for 99% IO without much going on) that you might 
have one of these circumstances.









=============================================================== From: Billy ------------------------------------------------------ Back in the day when servers didn=E2=80=99t have even 1 gig of ram, I had a m= achine having performance issues once it=E2=80=99s syslog grew above 10 gigs= ... Xfs is amazing.. --b te: ittle tiny log files in a directory...1 per event, timestamp-ish filename. roduction. After umpteenbazillion files (unknown) have been written, in ONE d= irectory.... the journaling file system gets really bonkers. tc. but in practice: Don't go there. his...) for a very long time. y slow (check iotop for 99% IO without much going on) that you might have on= e of these circumstances.