@dominic in #patchbay-dev

patchbay@6 perf

Hmm, did some more performance experiments!
Patchbay@6 loads only 30mb of heap, but if I disable /private and /notifications tabs, only 20! Disabling names and avatars maybe shaves another mb off that. Not loading the /public gets you down to around 15 and a empty electron app still loads 5. So patchbay load@6 is pretty damn light!

@dominic in #patchbay-dev
Re: %SPm+qoqzh

Okay! applying the same technique to patchbay@7 lopping off bits to see how light it can get. Removed patch-hub, patch-gatherings, ssb-horcrux, ssb-context (these didn't make much difference, but also don't show anything at first load) then parts of patchcore: contacts/obs about/obs channels/obs (just returned before they did their queries) and I got it down to 31 mb! There isn't much left, but there is still enough to write this message.

@dominic in #patchbay-dev
Re: %SPm+qoqzh

I tested %minbay and it lives up to it's name because stock minbay only loads a 22mb heap! (am running in electron for a fair comparison)

It doesn't use a tabs across the top either, and the side panel shinks... so it's mostly viable on a narrow screen already. if the side pane could be collapsed then even better.

@ev what sort of computer are you running this on? (and how much ram, etc?)

@dominic in #patchbay-dev
Re: %SPm+qoqzh

Tried patchwork - it weighs in at 97 mb heap, basically the same as patchbay@7, they both depend on patchcore so I'd expect them to be about the same.

@dominic in #patchbay-dev
Re: %SPm+qoqzh

Hmm... but since patchfoo uses zero javascript it only loads 5.6 mb (again, running in electron for a fair comparson) @cel what browser do you use?

@dominic in #patchbay-dev
Re: %SPm+qoqzh

okay, I couldn't figure out how to mention @cel correctly. But, maybe @ansuz's patchquery will solve those things?

@ansuz in #patchbay-dev
Re: %SPm+qoqzh

Patchquery? Did I write some code in my sleep again?

@mixxx in #patchbay-dev
Re: %SPm+qoqzh

you solved it, the answer ! now you just need to find where you put it

@dominic in #patchbay-dev
Re: %SPm+qoqzh

raw ssb-client, heaps 10 mb:

require('ssb-client')(function (err, sbot) {
  sbot.whoami(function (err, who) {
    var pre = document.createElement('pre')
    pre.textContent = JSON.stringify(who)
    document.body.appendChild(pre)
  })
})

there is actually quite a lot going on here, shs, muxrpc, etc. If this just used http it would be closer to 5mb I guess (patchfoo style)

@cellular in #patchbay-dev
Re: %SPm+qoqzh

@dominic i am using dillo.

a fair comparison might be to run patchfoo server in electron too.
server.js

mentions in patchfoo are done by using a @name mention and then entering the id in the text field for the name that appears after pressing preview. it will be pre-filled with an id if an id was mapped to that name recently. maybe now that there is ssb-about i will try making use of that to assist with picking the id.

@Matt in #patchbay-dev
Re: %SPm+qoqzh

Speaking of performance, I just completely rewrote Patchwork's rollup system to use a standard stream (with no windowing) using some ssb-backlinks trickery:

https://github.com/ssbc/patchcore/blob/master/feed/pull/rollup.js

I am now also caching the recent channels and using ssb-backlinks for mentions.

Load time has decreased by.. uh.. quite a lot! So snappy!

Next up, I want to rewrite message view to use ssb-backlinks.

@cellular in #patchbay-dev
Re: %SPm+qoqzh

@mmckegg i think you could use pull.unique() for the alreadyEmitted stuff. e.g.:

    return pull(
      pull.map((msg) => api.message.sync.root(msg) || msg.key),
      pull.unique()
    )

also, i think there is a typo in comparing rootMessage.key === rootMessage.key in AddReplies

@ev in #patchbay-dev
Re: %SPm+qoqzh

@dominic cool!

I'm running %minbase / %minbay on a Thinkpad S431 with 4 gigs of ram. That's not necessarily a slow computer, I just want to keep things fast and it's easy to notice with top when a program is running up memory. Now that flume is merged the whole sbot/minbase stack seems to be running fast and lite.

minbase.png

@Matt in #patchbay-dev
Re: %SPm+qoqzh

Ah, thanks @cel!

Good catches! Will update.

@dominic in #patchbay-dev
Re: %SPm+qoqzh

@Matt confirmed. All the names have appeared within 10 seconds, and the CPU has settled (using timeline feature) after 15. patchbay@7 is even better now too. But still not as good as minbay!

@ev I'm on a (second-hand) thinkpad x201. But it's about the same specs as yours. However, if I don't keep a eye on memory use that fills up and if it starts swapping it becomes so unresponsive it's quicker to power cycle it. I can't even get to a terminal to kill chromium or electron (the usual suspects). At least windows had ctrl-alt-delete. I don't really know how to fix this. I'm currently just keeping top running an checking memory use occasionally.

@neftaly in #patchbay-dev
Re: %SPm+qoqzh

Can you xkill from awesome?

@dominic in #patchbay-dev
Re: %SPm+qoqzh

that might work (would have to mess around setting up a keyboard shortcut) but that is treating the symptom not the disease. When it really gets stuck, I can't even move the mouse, completely unresponsive. If I could tell it not to prioritize one process so much that the window manager gets completely blocked? then I could still kill the program normally?

@noffle in #patchbay-dev
Re: %SPm+qoqzh

However, if I don't keep a eye on memory use that fills up and if it starts swapping it becomes so unresponsive it's quicker to power cycle it. I can't even get to a terminal to kill chromium or electron (the usual suspects).

@Dominic I keep having this happen on my x140e. I bet binding a hotkey to pkill -9 chromium would save me in 90% of cases!

@Mikey in #patchbay-dev
Re: %SPm+qoqzh

@Dominic remember %Hv2USdQ...?

@ezdiy in #patchbay-dev
Re: %SPm+qoqzh

Why linux VMM sucks so much

When system enters swap trashing spiral - typically some ill conceived buffering where the delays induces the offender proccess to touch even MORE memory, in a runaway spiral, everything gets paged out of memory and almost all processes spend time waiting on disk io.

kill -9 of the offending is not that helpful because:

  • you don't even have a way to issue that command. All already running processes are starved for RSS, and you think you can just spawn a new one willy nilly?
  • bunch of other, smaller processes enter mini-version of the runaway fibrilation of their own, where they queue a lot of pageins/outs IO because their RSS becomes too small for what they need.

Worst of all, once kernel sends off all the pages in dozens of idle processes to IO because javascript decided to ruin your day, this operation needs to finish, even if it becomes superfluous (ie memory is freed elsewhere).

So the only real recourse is:

  • kill offender as well a whole bunch of stuff and hope it will ease RSS pressure enough to stop the cascade
  • hope the victims don't have deep IO queue submitted because that makes the kill far less effective
  • do it from kernel, or something similiarly mlock()ed

That's what Alt+prtscr+k is for. It's a key for "kill X, everything in it, except tmux and make system responsive again". Most of the time anyway.


"Proper" way to do this is disabling overcommit and setting relatively small swap. But it's not really an option because the problem is not VM by itself, but linux RSS suckage. You can however tune overcommit ratio to strike the balance where runaway swapping becomes rare because the offender gets killed by OOM first, while still having wiggle room of moderate swapping.

@dominic in #patchbay-dev
Re: %SPm+qoqzh

@ezdiy but why can windows have ctrl-alt-delete bring up the task manager and that seems to mostly work? (tbh, I have barely touched windows for a decade, though)

@ezdiy in #patchbay-dev
Re: %SPm+qoqzh

@Dominic Task manager (well, that whole session) is more or less mlocked().

In addtition, NT VMM is derived from BSD, which doesn't starve the system equally under pressure, but it makes meaningful picks. In particular, NT kernel estimates RSS limits per process, and only past that threshold it starts swapping. This mechanism is much finer granularity than VM limits - vm can be any size, and it doesn't say anything about RSS the process actually needs.

Back in the day when RAM was fairly expensive, people ran busy BSD lamp servers mostly because it did so much better job than linux under memory pressure. This edge is kind of lost with current RAM sizes.

It should be noted that modern linux implements sensible active memory set scheduler, but only as a cgroup - it's all explicit, AFAIK, no sensible heuristics dealing the limits to reduce runaway swapping exist.

@ev in #patchbay-dev
Re: %SPm+qoqzh

@dominic I'm not using swap memory over here. I wonder if your hard drive is slow (is it an ssd?) or it could be damage? My Thinkpad has an ssd, so that speeds things up a bit.

@dominic in #patchbay-dev
Re: %SPm+qoqzh

@ev it's a spinning disk. Not gonna upgrade that it would go too fast, and I'd get lazy and flume would become slow.

@cellular in #patchbay-dev
Re: %SPm+qoqzh

happens to me lots too. i thought about making a program to kill -9 whatever process is using the most memory, but never got around to it. maybe i'll give a BSD another try one of these days

@dominic in #patchbay-dev
Re: %SPm+qoqzh

@cel okay I made this one: %killfree it's pretty crude, but i'm gonna test it. Kill chromium, firefox, and electron when memory usage gets to 70% (started at 90% but got stuck in swappsville without the script working, maybe that was just debugging, anyway this is about saving time so don't wanna waste too much time debugging it)

@cellular in #patchbay-dev
Re: %SPm+qoqzh

@dominic nice, i'll try that

@Kas in #patchbay-dev
Re: %SPm+qoqzh

For single apps, the easiest could be to use a wrapping script that set various ulimit values. When I use this method I usually give the script the same name as the original app and then put in earlier in $PATH. E.g.,

#!/bin/bash
# This is ~/.local/bin/chromium

# restrict total cpu time to 1 hour
ulimit -t 3600
# restrict rss size to 1 GB
ulimit -m $((1 << 20))
# see ‘help ulimit’ for more knobs

exec /usr/bin/chromium "${@}"

# eof

There are also more general approaches. This one is meant to be run by cron every minute: http://blog.endpoint.com/2012/08/automatically-kill-process-using-too.html

@ezdiy in #patchbay-dev
Re: %SPm+qoqzh

@kas Your script doesn't prevent the swap trashing problem, because it's not process group aware (chrome by itself is process group of tabs). It would also kill about 1/3 of webpages which eat up 1GB of VM innocently, as a matter of normal operation (from perspective of ulimit, my current chrome session uses about 70GB RAM). However if you have the luxury of having launch wrapper, go all the way, and use a cgroup!

On boot, create the memhog pen, and shove all your memhogs there:

cgcreate -g memory:/memhogs`
echo $((4096*1048576)) > /sys/fs/cgroup/memory/memhogs/memory.limit_in_bytes
echo $((512*1048576)) > /sys/fs/cgroup/memory/memhogs/memory.memsw.limit_in_bytes

And in your wrapper script:
exec cgexec -g memory:myGroup chromium

Setting up cgroup properly can pretty much fix most of these issues, the only downside is that it's not an automatic heuristic in kernel - something in user space needs to spell out rules for all the usual suspects, and even then you occasionaly get something you didn't anticipate (typically, terminal children are unrestricted, and you run node.js some script and bam...).

There are bunch of things which can manage cgroup automatically (cgmanager...), I have no idea how useful those are.

On BSD, there's rctl which can roughly accomplish the same, but is somewhat trickier to use memoryuse+vmemoryuse+swapuse can induce trashing instead, if you're not careful!

@Dominic Your idea is good (when running low on memory, kill all the usual suspects), but you need to count cached as free:

free | head -n 2 | tail -n 1 | awk '{print(($4+$7)/$2)}'

Otherwise something as simple as writing a large file quickly will kill your browsers.

Alternatively you can just drop the caches, echo 1 > /proc/sys/vm/drop_caches before you try the killing. This will slow the system down for a little while because everything is in writebcack/refetched, but is surprisingly still worth it because Linux swapd shows such exceptionally poor judgement about which pages to keep hot under pressure - it's kicking of wrong pages from cache in large part contributes to the page trashing problem in the first place.

@Kas in #patchbay-dev
Re: %SPm+qoqzh

@ezdiy,

go all the way, and use a cgroup!

That is an area I haven't touched yet, and it certain looks like it has potentials. Thanks for bringing it to my attention!

@dominic in #patchbay-dev
Re: %SPm+qoqzh

@ezdiy thanks! I didn't know about cgroups! I'll update my script for cached, too.

@cellular in #patchbay-dev
Re: %SPm+qoqzh

free | head -n 2 | tail -n 1 | awk '{print(($4+$7)/$2)}'

free | awk 'NR==2 {print(($4+$7)/$2)}'

:neckbeard:

@dominic in #patchbay-dev
Re: %SPm+qoqzh

Nice side effect of this script is you also get a tab amnesty. If I forgot what was open it can't have been too important!

@dominic in #patchbay-dev
Re: %SPm+qoqzh

I'm really enjoying running this kill script.
When chrome randomly closes it's like having master splinter pour a bucket of cold water on you.
Why was I looking at that anyway?

@ev in #patchbay-dev
Re: %SPm+qoqzh

@dominic does master splinter pour cold water on %minbase?

@dominic in #patchbay-dev
Re: %SPm+qoqzh

It generally happens when I'm using chromium, and usually looking at something stupid.

@ev in #patchbay-dev
Re: %SPm+qoqzh

master splinter just needed to pour some water on @gb's browser, as it started some sort of live video feed without her permission and just kept talking and talking about 'entrepreneurship' and 'influencers'.

@gb in #patchbay-dev
Re: %SPm+qoqzh

Yup. That happened. You laughing at the whole thing was water enough. Maybe a trickle but water enough. Back to the code! You ready?

@ev in #patchbay-dev
Re: %SPm+qoqzh

@gb Ready until tomorrow at 3:25pm.