[{"content":"GNOME has a wonderful feature of grouping applications in the Alt+Tab menu. Pressing Alt+Tab lets you quickly switch from Terminal to Firefox, without having to iterate through each of your multiple Terminal or Firefox windows.\nThe issue is that not all Terminal windows are equal. Perhaps you\u0026rsquo;ve got specific dev tools you want separated from other Terminals. In my case, I use mutt for email, and I want to be able to quickly Alt+Tab to my email without having to sift through a half-dozen terminals I\u0026rsquo;m using for other things.\nThink of it like when a browser lets you install a website as an \u0026ldquo;App\u0026rdquo; \u0026ndash; You can Alt+Tab directly to a GMail window, instead of having to find a specific browser tab in a specific browser window. GMail also gets it\u0026rsquo;s own icon on your task bar.\nBrief History I\u0026rsquo;ve written about this before, but the short version is:\nMost GNOME software does this simply with --class and --name for older X11 applications, and --gapplication-app-id for newer GNOME wayland applications (I\u0026rsquo;m sure KDE has similar args, but I\u0026rsquo;m not using that) gnome-terminal \u0026lt; GNOME 3.8 requires some extra steps to bypass the client-server model used to manage terminal processes (--disable-factory) gnome-terminal \u0026gt;= GNOME 3.8 switched to using some sortof dbus activation, which required additional hacks to configure a new DBUS service gnome-terminal at some later date (in late 2016) switched to also using a systemd service that needed to be set up At some point in the last few releases, the above stopped working again, and I\u0026rsquo;m tired of troubleshooting.\nThough I\u0026rsquo;ve generally been happy with gnome-terminal for the last 25 years, it\u0026rsquo;s probably about time to look elsewhere for a working solution.\nPtyxis Ptyxis is a newer terminal. It has some container features I haven\u0026rsquo;t investigated (due to not using containers, currently). I guess Ubuntu uses it by default now (but I\u0026rsquo;m using Fedora). You\u0026rsquo;ll need to figure out how to pronounce Ptyxis, as well.\nBut perhaps most importantly, it\u0026rsquo;s really easy to change it\u0026rsquo;s app-id.\nNote: My example is using a system-installed version of ptyxis. If you\u0026rsquo;ve installed via flatpak, you may need to adjust the Exec line appropriately.\nHere is the .desktop file for my mutt mail:\n$ cat ~/.local/share/applications/org.gnome.Ptyxis.pim.desktop [Desktop Entry] Name=Mail \u0026amp; PIM Comment=Mutt and Calendar Keywords=mail;mutt;calendar Exec=ptyxis --standalone --gapplication-app-id org.gnome.Ptyxis.pim --execute \u0026#34;screen -DR PIM -c ~/.screenrc-pim\u0026#34; Icon=mail-send Type=Application StartupNotify=true Some key details:\nProviding a --standalone flag to separate it from any existing ptyxis sessions Providing a custom --gapplication-app-id. This needs to match the name of the .desktop file to match correctly Things get funny if you don\u0026rsquo;t have periods in this, hence why I just added .pim to Ptyxis\u0026rsquo; default app-id. custom command with --execute And, most importantly: THAT\u0026rsquo;S IT.\nNo custom dbus services\nNo custom systemd units\nJust run the \u0026ldquo;Mail and PIM\u0026rdquo; application, and it\u0026rsquo;s got it\u0026rsquo;s own icon separate from your Terminal.\nWhat does it look like The actual application looks like mutt in a terminal, so no need to show that.\nThe Alt+Tab menu shows the separate app entry, though:\nAdditionally, it has it\u0026rsquo;s own icon in the dock, like the above.\nGNOME Terminal 🫡\n$ sudo dnf remove gnome-terminal GNOME Console To be honest, there\u0026rsquo;s another new GNOME Terminal: Console.\nI\u0026rsquo;ll be honest, I didn\u0026rsquo;t spend a whole lot of time looking at it for a very silly reason: I\u0026rsquo;ve got over two decades of typing \u0026ldquo;T\u0026lt;Enter\u0026gt;\u0026rdquo; to run a Terminal.\nFuture Changes It actually looks like things could possibly get even easier in the future. There\u0026rsquo;s a freedesktop proposal that includes app-id. I\u0026rsquo;m not sure if it will get merged, but if it does, this could make my \u0026ldquo;PIM\u0026rdquo; (theoretically) run in your default terminal, instead of ptyxis specifically.\nAlthough it looks like little movement has happened on that spec.\nPtyxis is compliant with this spec (how I found it), though I\u0026rsquo;m not sure how to actually use it yet.\n","permalink":"/posts/alt-tab-with-ptyxis/","summary":"\u003cp\u003eGNOME has a wonderful feature of grouping applications in the Alt+Tab menu. Pressing Alt+Tab lets you quickly switch from Terminal to Firefox, without having to iterate through each of your multiple Terminal or Firefox windows.\u003c/p\u003e\n\u003cp\u003eThe issue is that not all Terminal windows are equal. Perhaps you\u0026rsquo;ve got specific dev tools you want separated from other Terminals. In my case, I use \u003ccode\u003emutt\u003c/code\u003e for email, and I want to be able to quickly Alt+Tab to my email without having to sift through a half-dozen terminals I\u0026rsquo;m using for other things.\u003c/p\u003e","title":"Alt Tab With Ptyxis"},{"content":"I had firefox get OOM-killed today, which sent me down a rabbit hole of investigating why it was killed.\nObviously I was out of memory. But after firefox was killed, the system only went down to ~50% memory usage.\nThis didn\u0026rsquo;t turn out to be a typical page cache issue.\nInvestigation I\u0026rsquo;m not new to this, I know about buffers \u0026amp; caches. However, the output of free actually showed very little in either category. free (and htop) indicated I was using just under 7GB currently, although I could only account for about ~2GB of usage manually.\nI did the old-trusty drop pages:\n$ echo 1 | sudo tee /proc/sys/vm/drop_caches Indeed, I was still at about the same usage. Thinking there must be a program hogging that memory (in a way invisible to top?), I started digging.\nI shut everything down, logged out, switched to a TTY, and started disabling services, and killing processes. That freed up a few hundred megs, but I seemed to be stuck. Basically nothing was left except some kernel threads.\nOf note, this laptop is a ryzen with integrated Radeon GPU, and like any integrated graphics, it uses system memory for the GPU. However, I have that set to 1GB in the BIOS, and is actually pre-subtracted from my ram total (so the system shows 7/15GB used, instead of 7/16GB).\nNevertheless, I decided to see what radeontop showed, since GPU memory usage is kind of invisible to the tools I\u0026rsquo;d been using (at least in htop).\nradeontop showed very low VRAM usage (\u0026lt;100M of 1GB total), which makes sense, as any GPU-stuff was stopped.\nIt also showed very low GTT usage (\u0026lt;100M of ~7GB total), which was interesting, as what the hell even is GTT?\nGTT GTT is also referred to as GART, and is a method to use IOMMU to map system RAM to a GPU. It was a feature introduced with AGP graphics cards, and added to Linux during the 2.4 cycle. So.. yeah\u0026hellip; Lets pretend it\u0026rsquo;s new, then use that excuse to explain why I didn\u0026rsquo;t know what it was.\nIf your system is low on VRAM (only 1GB for my system) and you have free system RAM, the GPU can ask to steal some system RAM to work with. Neat. It\u0026rsquo;s probably not even slower on this system, since the integrated GPU\u0026rsquo;s VRAM is just system ram anyway.\nAll of this is fine\u0026hellip;\nSlabs The kernel allocates this GTT using something called the kernel slab allocator: It seems this is an optimization, to avoid having to do repeated malloc/free/malloc cycles (slow), the kernel will hold the slab of memory and re-use that for future needs.\nAnd there\u0026rsquo;s where my hidden usage was.\nThe GPU says it wants to use ram in the GTT The kernel allocates a bunch of ram to the GTT The GPU frees that RAM in the GTT There is no step 4. The kernel still holds this slab, in case the GPU needs it again. It\u0026rsquo;s effectively invisible in this state (not tied to a process, and usage is low in radeontop). That said, this is probably not actually a problem:\nWhen firefox was OOM-killed, that GTT would have still been in-use by the GPU, so it couldn\u0026rsquo;t exactly be freed. And it probably would be freed in the current state if memory pressure required it to be, such as another process allocating a bunch of ram). At the moment, we\u0026rsquo;re in a \u0026ldquo;good\u0026rdquo; state with lots of free memory, so no need to reclaim that unused slab.\nSo me seeing it allocated and unused is more likely just due to coincidental timing, rather than \u0026ldquo;GPU steals RAM and doesn\u0026rsquo;t give it back\u0026rdquo;.\nRegardless, it turns out you can poke the kernel to drop unused slabs in basically the same method used to drop the page cache (and now I know what the numbers are for):\n# Cache only $ echo 1 | sudo tee /proc/sys/vm/drop_caches # Slabs only $ echo 2 | sudo tee /proc/sys/vm/drop_caches # Slabs \u0026amp; cache together $ echo 3 | sudo tee /proc/sys/vm/drop_caches After using \u0026ldquo;3\u0026rdquo;, the system regained ~4.5GiB of RAM. Wow!\nWHAT WAS THE ISSUE So I suppose slabs are probably not a bad implementation.\nMy main concern is why does it take so much VRAM to display Firefox?\nMEASURING THE GTT I\u0026rsquo;m not sure if this is a Firefox, GNOME, or Wayland issue at this point, so I decided to do some tests. I measured an the following scenarios:\nempty desktop 10 new-tab firefox windows those same firefox windows maximized (2880x1800 at 100% scaling) Window Managers GNOME (Wayland)\nBefore (MiB) 10 windows (MiB) 10 maximized (MiB) VRAM 382 858 851 GTT 33 1118 2046 Weston (Wayland)\nBefore (MiB) 10 windows (MiB) 10 maximized (MiB) VRAM 346 862 833 GTT 46 954 1796 Blackbox (X11)\nBefore (MiB) 10 windows (MiB) 10 maximized (MiB) VRAM 311 814 808 GTT 37 518 1349 wlmaker (Wayland)\nI was actually really excited to try this, but it crashed in several of the tests. Mentioned purely because Window Maker was great, and I miss it.\nApplications So now the question is whether this was specific to Firefox. (\u0026ldquo;Before\u0026rdquo; usage here is a bit higher as this wasn\u0026rsquo;t a fresh login.)\nGedit on GNOME (Wayland)\nBefore (MiB) 10 windows (MiB) 10 maximized (MiB) VRAM 457 512 841 GTT 153 151 155 Similar behaviour occurred with gedit, though to a much lesser extent. It didn\u0026rsquo;t seem to need to request more GTT in this scenario, being mostly happy in VRAM.\nGnome Text Editor on GNOME (Wayland)\nBefore (MiB) 10 windows (MiB) 10 maximized (MiB) VRAM 546 969 948 GTT 158 229 799 Significantly worse than gedit, but not as bad as Firefox.\nObservations This behaviour is not GNOME or Wayland specific. GTT allocation increased as window size increased Blackbox is not a compositing window manager, so this probably isn\u0026rsquo;t the window manager\u0026rsquo;s fault Inspecting the GTT So we\u0026rsquo;ve seen VRAM/GTT usage climb based on number of windows, and size of those windows. The problem is that we can only tell as we\u0026rsquo;re observing it in real-time in a test scenario.\nFirefox in all of the examples above only ever reported about 900MiB RES via top, while it was actually consuming closer to 3000MiB of RAM between RES and the GTT. There doesn\u0026rsquo;t seem to be a standard way to list per-process VRAM use.\nFor amdgpu-supported GPUs, I was using radeontop, but that only gives a total VRAM and GTT number, there\u0026rsquo;s no breakdown. Another tool, umr, looks like it has the capability, though the CLI didn\u0026rsquo;t work for me at all. The umr GUI mode was initially looking helpful, though hat output simply shows firefox using 2MB, gnome-shell using 1MB, and \u0026ldquo;kernel\u0026rdquo; using 3GB. Not the smoking gun I was hoping for\u0026hellip;\nIf you\u0026rsquo;re in a RAM-constrained situation and GTT seems to be an issue, you\u0026rsquo;ll probably just have to make your best guess and close whatever you hope makes the biggest change. It\u0026rsquo;s probably whatever has the most and/or largest windows open.\nCONCERNS My concern at this point is that, as far as I\u0026rsquo;ve found, there\u0026rsquo;s no real way to identify VRAM or GTT usage.\nIt\u0026rsquo;s not consistenly possible to identify RAM is allocated to the GTT, but the GTT isn\u0026rsquo;t using it.\nI didn\u0026rsquo;t get into all the output of free from each test, but the GTT increases are not completely reflected by increases in buff/cache. man free says \u0026lsquo;cache\u0026rsquo; includes slabs, but apparently that\u0026rsquo;s not always true.\nMaybe actually allocated is not needed, as long as we can see proper usage and assume memory pressure will free the rest.\nI can\u0026rsquo;t see how much VRAM is used by any particular process, and top kind of lies by omission about RAM usage\nTop can tell you firefox is using 900MiB of system ram, but it doesn\u0026rsquo;t tell you it\u0026rsquo;s also using 2000MiB+ of VRAM/GTT. You can see total VRAM \u0026amp; GTT, but you just have to guess as to what\u0026rsquo;s caused that, and in which proportions.\nOn an 8GiB system (with 7GiB usable), that\u0026rsquo;s the difference between Firefox using 12%, and it\u0026rsquo;s real 40% figure. That\u0026rsquo;s a significant difference.\nAnd if I understand correctly, even a real GPU could still fall back to GTT and system ram, particularly if you have limited VRAM GPU. So this isn\u0026rsquo;t necessarily an iGPU issue, though it\u0026rsquo;s probably more relevant to those.\nWORKAROUNDS So now I started to look at limiting GTT allocation. According to dmesg, it could allocate up to 7GB of my RAM to the GTT, basically giving the GPU 50% of my 16GB!\namdgpu 0000:c3:00.0: amdgpu: amdgpu: 1024M of VRAM memory ready amdgpu 0000:c3:00.0: amdgpu: amdgpu: 7306M of GTT memory ready. Capping the total GTT memory limit. GTT limits are set by the TTM module (don\u0026rsquo;t forget to rebuild initrd):\n$ cat /etc/modprobe.d/ttm-gtt-limits.conf #Uncomment desired GTT limit #For 1GB: #options ttm pages_limit=262144 #options ttm page_pool_size=262144 #For 2GB: #options ttm pages_limit=524288 #options ttm page_pool_size=524288 #For 3GB: #options ttm pages_limit=786432 #options ttm page_pool_size=786432 However, this made my system unusable. While the GTT was indeed capped, GNOME became unresponsive once firefox was running.\nDisable hardware accelleration in Firefox. There\u0026rsquo;s a user-facing setting for this (under Performance).\nUnfortunately, Firefox is slooooow with this disabled. Scrolling is choppy, video seemed to play fine, but there was a delay while trying to interact with it (to pause, for example). Basically, I\u0026rsquo;m not using it like this.\nSo I\u0026rsquo;m back to where I was before, un-capped GTT and Firefox taking up all my ram (and lying about it).\nFUTURE INVESTIGATION Is there a way to see per-process VRAM usage? Can umr be fixed?\nI can see totals, but I can\u0026rsquo;t see that xGB is related to vlc or yGB is firefox\nI need a better way to manage \u0026ldquo;For Later\u0026rdquo; tabs.\nThis entire event wouldn\u0026rsquo;t have happened if I didn\u0026rsquo;t have 181 tabs.\nGranted, they are mostly in collapsed groups (and therefore most should be unloaded especially after restart), but this firefox profile still caused the GTT to use \u0026gt;4.5GB almost immediately, even though 171 tabs were not focused (and thus not loaded).\n","permalink":"/posts/where-is-my-ram/","summary":"\u003cp\u003eI had firefox get OOM-killed today, which sent me down a rabbit hole of investigating \u003cem\u003ewhy\u003c/em\u003e it was killed.\u003c/p\u003e\n\u003cp\u003eObviously I was out of memory. But after firefox was killed, the system only went down to ~50% memory usage.\u003c/p\u003e\n\u003cp\u003eThis didn\u0026rsquo;t turn out to be a typical page cache issue.\u003c/p\u003e","title":"Where is my missing RAM in Linux? Your GPU stole it!"},{"content":"I recently had the privilege to drive a 2024 MX-5 RF (\u0026ldquo;ND3\u0026rdquo;) for a week, thanks to Mazda Canada and my local Miata group: Miataphiles of London. It was an interesting opportunity to compare with my (slightly modified) 2016 MX-5 (\u0026ldquo;ND1\u0026rdquo;).\nTable of Contents Specs and Summary Non-Generational Cosmetics Modified vs Stock Soft Top vs Retractable Fastback ND1 vs ND2 ND1 vs ND3 Infotainment ND3 Technology Conclusion Specs and Summary A brief rundown of the two cars I\u0026rsquo;m comparing:\n2016 2024 Roof Soft Top Retractable Fastback HP 155 @ 6000 rpm 181 @ 7000 rpm Torque 148 lb-ft @ 4600 rpm 151 @ 4000 rpm Redline 6800 rpm 7500 rpm Trim GS (\u0026ldquo;Club\u0026rdquo;) GT Package Sport (Brembo, BBS, Recaro, etc) Grand Sport (Brembo, BBS, leather) Suspension Koni Sport w/ RoadsterSport springs; Flyin\u0026rsquo; Miata Front Sway Bar Bilstein shocks Exterior Jet Black Mica Soul Red Crystal Metallic\u0026quot; Interior Black w/ Recaro Seats Sports Tan Nappa Leather Photos Non-Generational Cosmetics My ND1 is black with black interior, and that unfortunately is the worst colour combo in retrospect. Especially in an MX-5, where the exterior colour wraps around to the interior panels, it\u0026rsquo;s all just overly monotone. Maybe worst is too strong? Least best? It\u0026rsquo;s not bad, it\u0026rsquo;s just\u0026hellip; all the other options are better, IMHO.\nThe ND3 was red with tan interior which is super nice to look at, both inside and outside of the car. Sitting behind the wheel and looking down the hood was great in red. In black, it\u0026rsquo;s only really cool at night listening to Phil Collins.\nI guess this can be solved with a wrap, though.\nLeather seats are a preference (and not mine \u0026ndash; I prefer cloth seats), but were very comfortable.\nModified vs Stock I\u0026rsquo;ve done a few changes to my ND1, swapping out the OEM Bilstein shocks and springs to Koni Sports with RoadsterSport Springs, as well as the Flyin\u0026rsquo; Miata Front Sway Bar (but not the rear). These springs in particular are roughly double the spring rate of the stock springs. This has had a few big effects:\nMuch rougher on the road. Especially bad roads. Even with the shocks at full soft (which makes it better \u0026ndash; but not great). As a driver, I can deal with this, but it\u0026rsquo;s not great as a passenger. It\u0026rsquo;s lost some of the \u0026ldquo;slow car fast\u0026rdquo; feeling. With the stock suspension, the car felt much faster than it actually was, allowing for a lot of fun within the speed limit. But to offset those points:\nMuch more composed on the track. Last Year vs This Year, I was two seconds faster \u0026ndash; despite being the first track day after the winter break, on my street tires (Pilot Sport 4S), on a cold damp day. Could I save two seconds on the stock suspension? Sure, probably. But these changes gave me so much more confidence as a starting point. I found ND3\u0026rsquo;s Bilsteins to be a little too bouncy over bumps on rough roads. But that\u0026rsquo;s better than being crashy like my ND1. It was nice to be able to A/B compare the two cars (sort of \u0026ndash; I obviously didn\u0026rsquo;t take Mazda\u0026rsquo;s corporate press car to the track). It also allowed me to compare ride-heights. As measured from the middle of the hub to the wheel arch (to avoid including tires in the figures):\nND1 Modified: 349mm ND3 Stock: 370mm Note that photos accentuate the wheel arch gap more than it appears in person. Unless you\u0026rsquo;re 60cm/24\u0026quot; tall, then this is probably how the car looks.\nND1 ND3 Excuse my extremely dirty car in the photos. It rained on the last track day, and I\u0026rsquo;ve been driving the ND3 all week.\nI think for a more track-focused car, I\u0026rsquo;m still happy with my changes. For a more road-focused car, maybe the Konis paired with the OEM springs (or comparable lowering spring) would be a better combination.\nThat said, the MX-5 is pretty great right out of the box. So there\u0026rsquo;s no need to rush in and modify things.\nSoft Top vs Retractable Fastback The RF is not made for me. It sucks, because (and this is entirely subjective) the RF is the better-looking car (at least with the roof up). It\u0026rsquo;s also the most \u0026ldquo;car\u0026rdquo; car, so if this is your only vehicle, or you drive year-round, the RF is probably the better choice.\nMy first exposure to Miatas was driving my in-laws RF years ago. I loved the car, but I didn\u0026rsquo;t fit very well. That was true then, and still true now. I\u0026rsquo;m 185cm/6'1\u0026quot; \u0026ndash; tall but not unusually so. However, no matter how I adjust the seat, my head wants be where the RF top is, unless I scooch my head down \u0026ndash; which is not comfortable in the long term, and still leaves my head touching the roof.\nI found there is a tad more room with the soft-top (and it hurts less on bumps when you hit it), though I had some neck issues a few weeks after getting mine. I finally realized I couldn\u0026rsquo;t see most stoplights without tilting my head. That problem was finally resolved with an Aurora Auto Design Low Profile Seat Mount. The bracket was a pain to install, hands-down the best QOL improvement I\u0026rsquo;ve made. I can even fit with my helmet on now. It feels like I\u0026rsquo;m sitting in the car vs on the car.\nI\u0026rsquo;m worried that with the RF, even with Aurora seat mount, it would still be too tight for me.\nWind Noise was definitely louder in the RF. It didn\u0026rsquo;t seem to make a big difference on windows down vs up, that just seemed to change where the wind noise was coming from. After some experiments with my hand near the mirror, I was able to significantly reduce wind noise. Some after-market wind foilers would probably be near the top of my mod list, if I owned the RF\nThat said, I\u0026rsquo;m speaking as somebody who isn\u0026rsquo;t overly attached to the concept of the convertible to begin with. I\u0026rsquo;d be quite happy with a MX-5 Coupé \u0026ndash; provided it had sufficient head room \u0026ndash; or at least a Gurney Bubble.\nND1 vs ND2 I don\u0026rsquo;t have a 2019 ND2 on me. However, there are some changes made in that year that persist into 2024, and I felt I should address them.\nThe higher redline made possible by the revised engine in 2019 is very-much appreciated. In my ND1, I often hit the rev-limiter in first doing hard pulls. Granted, I wasn\u0026rsquo;t taking the ND3 to the track or autocross, but I was able to time that shift much easier with the 700 RPM available. It sounds silly, but it makes a difference.\nI did not notice any performance difference on the street. I don\u0026rsquo;t know what I was expecting, but it really felt like the same amount of power and torque, it just revs a bit higher. That\u0026rsquo;s not a bad thing, but on paper, a 17% increase in power sounds more significant than a butt-dyno can detect. Again, I didn\u0026rsquo;t track the car, so maybe it would make a difference in times, if not feeling.\nThe telescoping steering column is a huge comfort improvement.\nND1 vs ND3 The LED lights are a nice improvement. I\u0026rsquo;m not a fan of fake vents, and the MX-5 has some as a result of this change. It will be convenient for those who wish to add brake ducting, though. I understand in Japan you can still option the separate DRLs. Hopefully that\u0026rsquo;s an option that comes here, too.\nThe steering seemed lighter \u0026ndash; but I never thought the 2016\u0026rsquo;s steering was particularly heavy, but my project car is a 1988 Pontiac Fiero with a manual steering rack. The steering did seem light, maybe that was \u0026ldquo;frictionless\u0026rdquo;, but how much of that was detectable vs placebo, I\u0026rsquo;m not sure.\nI\u0026rsquo;m not qualified to talk about the revised Differential, I didn\u0026rsquo;t get to use the new track-mode traction control, and I didn\u0026rsquo;t feel any effects of G-Vectoring control on the road.\nInfotainment I split this out from the rest of the Technology section because you\u0026rsquo;ll deal with this daily. The new infotainment is about a million times better.\nThe screen is brighter and easier to read in the day. It\u0026rsquo;s better at night. It\u0026rsquo;s a shorter form factor, so it stays out of your visibility better. It\u0026rsquo;s just all-around a significant improvement.\nThe hardware behind it is responsive. I didn\u0026rsquo;t touch Mazda\u0026rsquo;s software, instead exclusively using wireless Android Auto. It launched faster, and operated smoother than in my 2016, which is prone to occasional hiccups (and can often take 30 seconds to actually launch AA).\nWireless Android is particularly convenient in this car because there\u0026rsquo;s nowhere to stash a modern phone. Granted, you can get wireless AA with the older infotainment by buying a $100 AA Wireless Dongle, but now you don\u0026rsquo;t need a dongle.\nThat said, running Wireless Android Auto can (and will) destroy your phone battery. But at least when you need to plug in to charge, the ND3 has USB-C ports that put out 20W (9V 2A). The ND1 ports were limited to 10W (5V 2A) (if you had the upgraded Android Auto hub, at least).\nND3 Technology The big changes for 2024 are all technology related: All new vehicle electrical platform which allows for the new Infotainment (see above), and new features (automatic high-beams, speed limit detection, radar cruise, etc). All of these features have existed on other cars for years, but are new to both the MX-5 and to myself. I\u0026rsquo;ve never owned or driven a car with any of these features.\nStreet sign detection is pretty handy, especially when driving in unfamiliar areas. Apparently this uses a camera, and displays the current speed limit on the dash, as well as allowing a single button-press to set the cruise target to the current speed limit.\nThis worked rather well, and is much better than incomplete (or worse: outdated) databases from mapping providers. The only caveat is that, obviously, it requires a street sign to be present and visible for it to be read. So you\u0026rsquo;ll need to be aware what the unposted limit is, which can vary by municipality. That\u0026rsquo;s a minor detail for such a helpful feature, though. I also never found it giving me the wrong information. It was remarkably good at figuring out when I changed roads, and taking away the speed limit sign until it saw another one.\nAutomatic high-beams are another feature I\u0026rsquo;ve never used. It will automatically toggle on/off your high-beams when appropriate. I found this feature to be extremely annoying, as it spasmodically toggled them on/off during a night-time country drive. I think at times it was detecting reflective road signs as oncoming cars. I pulled over during my first night drive specifically to disable this feature and switch back to manual high-beams. Perhaps this can be fixed with software updates to the sensitivity.\nRadar cruise is something I was looking forward to the most, and left me with mixed results that I still haven\u0026rsquo;t come to a conclusion on. I had to pick up the car in Toronto, and drive 200km back home. I figured the multi-multi-lane 401 was a perfect use-case for Radar cruise. It wasn\u0026rsquo;t.\nI had set a comfortable follow distance to the car in front of me, and things seemed to be going well with minor changes in speed. Then a car merging on from the right wanted to cross through my lane, into the lane to my left, and I wouldn\u0026rsquo;t need to alter my speed. As a human with driving experience, I knew the intention, though obviously I can forgive a computer for not making that assumption. However, even if I was wrong and the driver stayed in my lane, there was sufficient space that I could just coast-down in-gear to allow some more room. No danger or reason to panic.\nHowever, as soon as the car started entering my lane, the radar cruise hit the brakes somewhat stronger than I expected. I played around with the cruise for a bit more, but wasn\u0026rsquo;t comfortable with it until I was well out of traffic (about an hour later).\nCommuting on provincial highways (traditional country two-lane highways) was a much nicer experience. I still find it was far too happy to speed up to cars and hit the brakes, rather than coasting-down from further away. But it performed as advertised, just not how I\u0026rsquo;d prefer it to be implemented.\nOne annoying effect of the cruise being able to control the brakes is that it will actively brake the car if you\u0026rsquo;re driving over set cruise speed. The best example I have is a large hill on my way home from work. It\u0026rsquo;s a fairly steep drop to a stream, followed by a hill climb. In any other car with non-radar cruise, the car will happily gain a bit of speed down the hill \u0026ndash; which is fine, because it will use it climbing the hill in a few seconds. The ND3, however, will brake the whole way down the hill, just to lay on the gas to get up the hill.\nAnother example was passing a slow car. Say you have your cruise set to 90km/h, and you come across a car doing 75km/h. You pull out and hammer it to pass, temporarily having your speed in excess of your cruise set speed. As soon as you let off the gas, the car will start braking to get you back down to 90 \u0026ndash; instead of letting the car coast down on it\u0026rsquo;s own.\nNow, I don\u0026rsquo;t expect the car to understand I\u0026rsquo;d like to preserve momentum on a hill. But I don\u0026rsquo;t think it\u0026rsquo;s unreasonable that the car should understand that if I hit the gas to go to a certain speed, I\u0026rsquo;m comfortable at that speed, and I\u0026rsquo;m fine letting the car coast down.\nAnother noteworthy change to cruise behaviour is Cruise doesn\u0026rsquo;t disengage when you hit the clutch pedal. So you can shift up/down, and the cruise control will continue once you\u0026rsquo;re in gear. This was very surprising to me, as I expect any clutch/brake press to immediately disable the cruise control.\nI can\u0026rsquo;t say this behaviour is wrong, but it\u0026rsquo;s very different from any car I\u0026rsquo;ve driven before. I can\u0026rsquo;t imagine a scenario where I want to switch gears, but can\u0026rsquo;t press \u0026lsquo;RES\u0026rsquo; to confirm I still want cruise active. That said, I haven\u0026rsquo;t settled my thoughts on the whole cruise control scenario. It does require driver adjustment, and I find some of it\u0026rsquo;s behaviours to be outside my comfort zone. But maybe my comfort zone would change after having used it for longer?\nConclusion My overall impression was how remarkably similar the ND1 is to the ND3 (aside from changes I\u0026rsquo;ve made). \u0026ldquo;Well Duh\u0026rdquo;. The 2024 MX-5 is a great car, but so was the 2023, and the 2022\u0026hellip; It would have been very easy for Mazda to change things up. I think the Miata is in a very unique position that changes might actually detract from how great this car is, as-is, straight out of the box \u0026ndash; For example, my changes made my car a much worse road car. So I\u0026rsquo;m not disappointed that it is very similar, but I am surprised and pleased Mazda has such restraint.\nIf you\u0026rsquo;re considering buying a newer MX-5, it\u0026rsquo;s absolutely worth considering for the Infotainment QOL improvements alone. If you just got your 2023 and regret it: It\u0026rsquo;s really not much different, and I don\u0026rsquo;t think there\u0026rsquo;s reason to fret.\nIf you\u0026rsquo;re like me, with an older ND, then the upgrade is questionable. I don\u0026rsquo;t see myself spending money to upgrade from my ND1 to an ND3. If something happened to my ND1, though\u0026hellip; A new ND3 would definitely have some bonus points over a used ND1/ND2.\n","permalink":"/posts/2016-ND1-vs-2024-ND3/","summary":"\u003cp\u003eI recently had the privilege to drive a 2024 MX-5 RF (\u0026ldquo;ND3\u0026rdquo;) for a week, thanks to Mazda Canada and my local Miata group: Miataphiles of London. It was an interesting opportunity to compare with my (slightly modified) 2016 MX-5 (\u0026ldquo;ND1\u0026rdquo;).\u003c/p\u003e","title":"Thoughts after a week with the 2024 MX-5 RF (ND3)"},{"content":"In addition to moving this blog to cloudflare, I decided to take advantage of enabling Cloudflare\u0026rsquo;s proxy at the DNS level for my other (non-blog) services. Basically, this makes Cloudflare act as a reverse proxy.\nHowever, handling Firewall \u0026amp; Port Forwarding rules was a bit more complicated.\nTable of Contents Why Easy Way: Port Forwarding Ports 80 \u0026amp; 443 Cloudflare IPs Creating Port/IP Groups in Unifi Network Firewall \u0026amp; Port Forwarding: IPv4 Firewall: IPv6 Conclusion Why First, why use Cloudflare proxy:\nThey have much more sophisticated attack detection and anti-bot monitoring than I ever could Some stuff will be cached fairly significantly, which helps since I\u0026rsquo;m hosting these services on a slower connection. Point #2 is a bit unnecessary, since these services are mostly used by me or family, so there shouldn\u0026rsquo;t be significant bandwidth concerned. However, #1 is important. While I can keep on top of patching and updates, I can\u0026rsquo;t keep on top of the latest bot or attack activity. Cloudflare does this, though. And with the flip of a switch I can disable the page at Cloudflare\u0026rsquo;s end (or add a captcha for access).\nEasy Way: Port Forwarding Ports 80 \u0026amp; 443 The easy way to do this is the same as any hosting internally:\nIPv4: Port Forward ports 80 \u0026amp; 443 to your internal web server IPv6: Add a firewall rule allowing ports 80 \u0026amp; 443 to talk to your web server\u0026rsquo;s IPv6 address (Rejoice: no NAT!) However, since my primary motivation in adding Cloudflare is for an extra layer of security \u0026amp; proxying, allowing connections to bypass that isn\u0026rsquo;t ideal.\nCloudflare IPs I spent a good several minutes looking for \u0026ldquo;Cloudflare proxy values\u0026rdquo;, \u0026ldquo;Cloudfare address ranges\u0026rdquo;, etc. Turns out \u0026ldquo;Cloudflare IPs\u0026rdquo; takes you where you want: A list of Cloudflare IP addresses. Also available in plain text for IPv4 and IPv6.\nCreating Port/IP Groups in Unifi Network If you\u0026rsquo;re not using a Unifi router, your configuration will, obviously, be different. That said, the same concepts probably apply.\nWithin the Network app, I navigated to Settings, then Profiles, then scrolled down to Port/IP Groups. You\u0026rsquo;ll need to create a few groups:\nA IPv4 group for all Cloudflare IPv4 IPs:\nWe\u0026rsquo;ll use this in the IPv4 firewall rules.\nAn IPv6 group for all Cloudflare IPv6 IPs.\nNo screenshot. Like above, but six-ier.\nYou can\u0026rsquo;t mix IPv4 \u0026amp; IPv6 in one list.\nWe\u0026rsquo;ll use this in the IPv6 firewall rule.\nAn IPv6 Group for your web server\u0026rsquo;s public IPv6 address.\nWe\u0026rsquo;ll use this in the IPv6 firewall rule.\nA Port Group with Ports 80 \u0026amp; 443 in it.\nWe\u0026rsquo;ll use this in the IPv4 \u0026amp; IPv6 firewall rules.\nFirewall \u0026amp; Port Forwarding: IPv4 You\u0026rsquo;ll still need to configure an IPv4 Port Forward. This is because NAT sucks, and the router still needs to do work to handle readdressing. Ugh.\nThis will open Ports 80 \u0026amp; 443 to the Entire Internet. This works (and I\u0026rsquo;ve been doing it for years), but is not my ideal end state with Cloudflare.\nNote that while you can change From: Any to Limited, that gives you a very small plain text box, which is annoying. You can probably enter multiple IP ranges? Maybe? I didn\u0026rsquo;t try, because a small text box is annoying and hard to maintain.\nI decided to implement an \u0026ldquo;Allow\u0026rdquo; rule for Cloudflare IPs, and a \u0026ldquo;Block\u0026rdquo; rule as a catch-all. This is because you can\u0026rsquo;t inverse a list of IP addresses (match things not in the list). So, two rules:\nAllow HTTP(S) from Cloudflare over IPv4\nThis explicitly allows Cloudflare to talk on ports 80 \u0026amp; 443.\nBlock HTTP(S) from Internet over IPv4\nThis rule will block all traffic over ports 80 \u0026amp; 443, from anywhere. But because the Cloudflare addresses were already matched, they still get in.\nThe ordering is important. You\u0026rsquo;ll want your Allow rule before the Block rule.\nFirewall: IPv6 As is so often the case, everybody worries that IPv6 is going to be so much harder, when the reverse is actually true. Only one Firewall rule is needed:\nNo Port Forwarding, no multiple rules, no order considerations. Just Done.\nConclusion Don\u0026rsquo;t forget to test with external hosts to ensure the rules behave as expected.\nAlso, you may want to follow-up with rules to restore the original (non-Cloudflare) request IP in your web server of choice. If you\u0026rsquo;re doing this, it\u0026rsquo;s probably important to not allow general Internet traffic, else a bad actor could spoof the CF-Connecting-IP header.\n","permalink":"/posts/cloudflare-proxy-with-unifi-firewall/","summary":"\u003cp\u003eIn addition to \u003ca href=\"cloudflare-pages-hugo\"\u003emoving this blog to cloudflare\u003c/a\u003e, I decided to take advantage of enabling Cloudflare\u0026rsquo;s proxy at the DNS level for my other (non-blog) services. Basically, this makes Cloudflare act as a reverse proxy.\u003c/p\u003e\n\u003cp\u003eHowever, handling Firewall \u0026amp; Port Forwarding rules was a bit more complicated.\u003c/p\u003e","title":"Cloudflare Proxy with Unifi Firewall"},{"content":"I\u0026rsquo;ve been struggling with Home Assistant notifications. From some searching, it seems other people are also struggling.\nA lot of videos online cover basic \u0026ldquo;How to send a notification to your phone\u0026rdquo;, but I wanted a whole lot more life-cycle than that. I want to clear them. And I don\u0026rsquo;t want to have to write a thousand action conditions, either.\nI sat down and spent far more time than I\u0026rsquo;d like to admit to tackling the problem. This might not fit every use-case for every person, but this was exactly the functionality I needed.\nTable of Contents Changes Goals Make Notifications Notifications from Automations Push/Sync Notifications with Android Phones Forward Notifications to Mobile Phones Dismiss notifications on Mobile Phones Dashboards Notifications in dashboard Without HACS With HACS Home Feed Card My Dashboard (Vertical) My Dashboard (Horizontal) Neat Tricks Caveats and Possible Improvements Super Helpful Tips Downloads Thanks Changes 2023-01-31: Added a Table of Contents, and added a few layers of nested headers to help organize. Goals My basic list of goals were as follows:\nNotifications that can be displayed in Home Assistant dashboard.\nI\u0026rsquo;m installing wall-mounted tablets. Notifications must be visible and apparent when walking past.\nNotifications that are also relayed to Mobile (Home Assistant app on Android).\nI may not be walking past a tablet (or even home) when a notification occurs.\nNotifications can be dismissed via automation\nMy theory of notifications are that they are states that need to be addressed, rather than notices to swipe away. Rather than having to dismiss the \u0026ldquo;Door Unlocked\u0026rdquo; or \u0026ldquo;Washer Finished\u0026rdquo; notification, the corrective action (Locking the door, opening the washer) should clear the notification. Both on the dashboard, and the phone.\nI was solving this with a dashboard cards that lit up when triggered. However, they were always there, even if not valid \u0026ndash; i.e., I always have an icon for a washing machine, even if it\u0026rsquo;s not helpful most of the time.\nNot needing to update every automation to adjust mobile notification rules.\nIf I add/change/remove/upgrade my phone, I don\u0026rsquo;t want to review rules. Ditto for adding my wife to notifications, etc.\nDo all of the above with NO HACS ADDONS, or at minimum, as few as possible\nI don\u0026rsquo;t have anything against HACS (and I use it). But I also don\u0026rsquo;t like building a house of cards and weird one-off special purpose cards. Core HA-included functionality should cope with HA upgrades, without issue.\nSpoiler alert: No HACS addons are needed for most of this, including the mobile notifications. However, I am using one HACS addon for the Dashboard. It technically isn\u0026rsquo;t required, but it significantly improves the experience. I\u0026rsquo;ll detail the dashboard options below.\nAnd as always, there are numerous ways to do anything in Home Assistant. This isn\u0026rsquo;t \u0026ldquo;correct\u0026rdquo; or \u0026ldquo;incorrect\u0026rdquo; \u0026ndash; merely \u0026ldquo;solves the intended scenario\u0026rdquo;.\nNote, there are downloads available at the end of this post.\nMake Notifications Notifications from Automations Automations create notifications\nUsing the persistent_notification.create service to create notifications allows you to create notifications that are handled by Home Assistant itself. Setting a notification_id is very, very important. If you call notify.persistent_notification instead, you don\u0026rsquo;t get to set that.\nFor example, via automation, when my Washing Machine power drops to 0.0W this Action is performed:\n- service: persistent_notification.create data: title: Washing Machine message: Load Complete! notification_id: Washing Machine Automations dismiss notifications\nUse persistent_notification.dismiss to dismiss notifications when the original status has been resolved. As I said earlier, I want notifications to be dismissed by resolving the condition that set it. This requires notification_id, so we can clear specific notifications.\nWhen I open my washing machine door, for example, this action is performed:\n- service: persistent_notification.dismiss data: notification_id: Washing Machine You can set up those notifications however you want: one automation that creates/dismisses based on an if/else, multiple automations with complex conditions. You may create a dozen automations creating/dismissing 100 notifications. That\u0026rsquo;s up to you. The key thing is that you\u0026rsquo;re creating/destroying them with the persistent_notifications service.\nPush/Sync Notifications with Android Phones Forward Notifications to Mobile Phones Here we can create a new automation to forward notifications to the Home Assistant app, installed on my device.\nWe can trigger an automation whenever the persistent_notification.create service is used. From there, we can create a mobile notification, and re-use the message, title, and notification_id from the original notification:\nBelow is the entire automation used to relay all notifications to my phone:\nalias: Notifications - Send To Mobile description: \u0026#34;\u0026#34; trigger: - platform: event event_type: call_service event_data: domain: persistent_notification service: create condition: [] action: - service: notify.mobile_app_pixel_6a data: message: \u0026#34;{{trigger.event.data.service_data.message}}\u0026#34; title: \u0026#34;{{trigger.event.data.service_data.title}}\u0026#34; data: tag: \u0026#34;{{trigger.event.data.service_data.notification_id}}\u0026#34; channel: \u0026#34;{{trigger.event.data.service_data.notification_id}}\u0026#34; mode: single You\u0026rsquo;ll need to add each mobile device you want to send notifications to (eg. Me, wife, etc). However, you don\u0026rsquo;t have to also do that on the dozens/etc automations for the Washer, Front Door lock, etc.\nThese don\u0026rsquo;t need to be mobile notifications, of course. This could relay to any notification channel (XMPP, etc).\nDismiss notifications on Mobile Phones The Home Assistant Companion App Notification Docs mention that you can actually dismiss notifications as well, by sending a specially-crafted message (clear_notification) along with, you guessed it, the notification_id.\nOne last automation to dismiss the notification from our phones whenever the persistent_notification is dismissed:\nalias: Notifications - Dismiss to Mobile description: \u0026#34;\u0026#34; trigger: - platform: event event_type: call_service event_data: domain: persistent_notification service: dismiss condition: [] action: - service: notify.mobile_app_pixel_6a data: message: clear_notification data: tag: \u0026#34;{{trigger.event.data.service_data.notification_id}}\u0026#34; mode: single Again, you\u0026rsquo;ll have to duplicate the action for additional devices.\nNow, while the notifications themselves could be sent to any notification provider, these clear_notification messages are specific to the companion app on Android \u0026ndash; The docs mention this doesn\u0026rsquo;t work on iOS.\nDashboards Notifications in Dashboard The built-in persistent notifications are not very noticeable in Home Assistant (little bell icon in the side bar gets a coloured dot on it). This was insufficient for me.\nHowever, we can put notifications right in the dashboards with two options:\nWithout HACS Without HACS, we can use an entity-filter card. This has a significant downside in that you need to pre-enumerate every notification_id you use. If you want this widget on multiple dashboards, adding a new notification is a significant chore.\nI don\u0026rsquo;t suggest actually using this, except as an example that it can be done\n- type: entity-filter entities: - persistent_notification.abc secondary_info: last-changed type: attribute attribute: message - persistent_notification.xyz secondary_info: last-changed type: attribute attribute: message - persistent_notification.zzz secondary_info: last-changed type: attribute attribute: message state_filter: - notifying card: type: entities title: Manual Filter Entities w/ Secondary You must use the entity-filter card with filter based for \u0026rsquo;notifying\u0026rsquo; state. This will exclude notifications that don\u0026rsquo;t currently exist. If we used a plain entities card, you\u0026rsquo;ll get errors for every notification not currently active.\nWith HACS The much better option is to use Auto-Entities via HACS. It provides some significant benefits:\nfilter includes, not just excludes. Additionally, we can use a wildcard. This makes this a generic card now. You can drop it on many dashboards without having to touch it again in the future.\nIt has a show_empty value, which makes the whole widget disappear if empty.\nIt has a empty value, which lets you select an alternate card to display if empty (vs show_empty, above)\nIt lets you use any container-type card (grid, vertical-stack, etc), combined with any card for the entities. This gives you much more style flexibility.\nYou can see why I compromised on my \u0026ldquo;No HACS\u0026rdquo; goal for this point.\nAnd this, much simpler yaml:\n- type: custom:auto-entities card: type: vertical-stack title: Notifications in Vertical Stack of Cards card_param: cards filter: include: - entity_id: persistent_notification.* options: type: entity attribute: message sort: method: last_triggered Gives us this much, much nicer list. Also note my Washing Machine notification is here. It wasn\u0026rsquo;t in the previous example because I hadn\u0026rsquo;t hard-coded it.\nHome Feed Card There\u0026rsquo;s also a Home Feed Card in HACS specifically to display persistent_notifications. However, it\u0026rsquo;s style didn\u0026rsquo;t match my dashboard. I may have grown to like it, but I\u0026rsquo;m very happy with the above solution.\nMy Dashboard (Vertical) Here\u0026rsquo;s a screenshot of my (still in progress) main dashboard, with notifications displayed in a vertical -stack of cards on the left. This gives me very visible notifications when present:\nWhen there are no notifications, that space is used instead for a weather forecast.\nI am using two frontend addons from HACS on this dashboard, however.\nlayout-card to gridify my dashboard but with grid layout and placement control.\nCanary, allowing some basic css styles (canary_style)\nCanary is not necessarily the best option for styles, but it is the one I was experimenting with.\nMy Dashboard (Horizontal) An alternative dashboard I\u0026rsquo;m considering is top-notifications that disappear entirely when clear:\nWhen there are no notifications, that space is reclaimed by the rest of the dashboard.\nNeat Tricks Since we set a channel on the Mobile Notification, we can actually long-press on the notification and adjust notification settings (on/off, silent, pop-up on screen) on your phone itself. These are configured per-channel.\nThis also means your notification_id values should be user-readable, since we re-used these for Channel IDs. Don\u0026rsquo;t use xyz3 like my test data, because these channels never go away once seen, and your notification settings will look like this:\nThis may also be Android-only. Check the docs.\nSince we set a tag on the Mobile Notification, re-sending a notification replaces the old one with the same tag, rather than giving you a second notification. I\u0026rsquo;m re-using notification_id here as well.\nCaveats and Possible Improvements It sucks to have to list every mobile device in the two forwarding automations. It would be nice to have some logic that can do a wildcard match and loop. Since I have only two phones to configure, but many notifications, this was good enough.\nYou could add key words to message/title/notification_id (ex: \u0026ldquo;Urgent:\u0026rdquo; or \u0026ldquo;- Mobile\u0026rdquo;), then add a condition to only forward notifications to phones if that key word is there.\nI may do this in the future. There\u0026rsquo;s some notifications that will be irrelevant unless you\u0026rsquo;re actually home.\nIf you manually dismiss a notification in Home Assistant, the notification is dismissed on mobile.\nHowever, if you \u0026ldquo;Dismiss All\u0026rdquo; from the Home Assistant sidebar, only one notification is actually dismissed on the phone. This may be a fixable problem, but that\u0026rsquo;s not a problem I\u0026rsquo;m concerned about since the side bar will probably be hidden on my wall tablets.\nSuper Helpful Tips There are a bunch of docs and tools I used to accomplish this:\nHome Assistant\u0026rsquo;s Developer Tools \u0026gt; Events Tab\nHelped clue me in that I could trap the service call when creating/dismissing persistent notifications\nLogbook\nMany log items are just notices that something updated with no further data. However, if you click on an \u0026lsquo;Automation\u0026rsquo; in Logbook, theres a whole lot of detail on what the automation did/didn\u0026rsquo;t do, as well as the data available to it at each stage.\nOnce I had the \u0026ldquo;Send to Mobile\u0026rdquo; notification triggering correctly (with test content only), I had to figure out which notification actually triggered the automation. I clicked on the Automation in the log, and inspected the first step (Executed).\nThere\u0026rsquo;s a tab in the bottom pane called \u0026ldquo;Changed Variables\u0026rdquo;. This identifies all of the data available for you that was set when the automation was triggered. This is how I was able to identify how to reference the message/title/notification_id data that was passed to the service call.\nNotification documentation for the Companion App (Explained tags/channels, and a ton of other features)\nhttps://companion.home-assistant.io/docs/notifications/notifications-basic/\nDownloads Here\u0026rsquo;s some examples you can download and shove into your HA to test.\ndashboard-notifications.yaml\nThe dashboards require some HACS addons as-is, but no actual customization is needed for them to display notifications. (Obviously, the buttons won\u0026rsquo;t work for you)\nautomation-send-to-mobile.yaml\nYou\u0026rsquo;ll need to adjust the service line to reference your phone\u0026rsquo;s notification service. And/or add additional actions for additional phones.\nautomation-discard-to-mobile.yaml\nYou\u0026rsquo;ll need to adjust the service line to reference your phone\u0026rsquo;s notification service. And/or add additional actions for additional phones.\nThanks I hope this will be helpful to others.\n","permalink":"/posts/homeassistant-notifications-sync-dashboard-mobile/","summary":"\u003cp\u003eI\u0026rsquo;ve been struggling with Home Assistant notifications. From some searching, it seems other people are also struggling.\u003c/p\u003e\n\u003cp\u003eA lot of videos online cover basic \u0026ldquo;How to send a notification to your phone\u0026rdquo;, but I wanted a whole lot more life-cycle than that. I want to \u003cem\u003eclear\u003c/em\u003e them. And I don\u0026rsquo;t want to have to write a thousand action conditions, either.\u003c/p\u003e\n\u003cp\u003eI sat down and spent far more time than I\u0026rsquo;d like to admit to tackling the problem. This might not fit every use-case for every person, but this was \u003cem\u003eexactly\u003c/em\u003e the functionality I needed.\u003c/p\u003e","title":"Home Assistant Notifications (with Sync in Dashboard and Mobile)"},{"content":"This is the first post after migrating this blog to cloudflare-pages hosted (pushed from gitlab.com). Why?\nThis blog is assembled using Hugo, a static site generator. This makes the server-side basically irrelevent. There\u0026rsquo;s no PHP, dotnet, Perl or Coldfusion that needs to run. The site is effectively \u0026lsquo;compiled\u0026rsquo; into it\u0026rsquo;s final form. So really, there\u0026rsquo;s no reason to pay $12/mo to host it on Linode. But I have been, for over a decade (granted, it was less money a decade ago).\nThe Linode VM, to be fair, also hosted other services: Mumble to voice chat with my friends (who have all moved to Discord), XMPP to text chat with\u0026hellip; well, nobody. A few other things I\u0026rsquo;m sure I forgot about as they\u0026rsquo;ve slowly bitrot over years of updates/lack of use/lack of care.\nSo the decision was made: Get something else to host my website, and ditch the Linode.\nThe obvious answer is gitlab.com\u0026rsquo;s \u0026lsquo;Pages\u0026rsquo;. And I\u0026rsquo;ve had a test build of this site on gitlab pages for a long time. The source for this site is hosted in a Gitlab.com repo, and has been for years (Previously, I used a self-hosted gitlab instance which worked well \u0026ndash; until a fairly minor issue I just didn\u0026rsquo;t care to spend time to fix killed it. It is effectively just a checkbox to have it build a hugo-based website, totally automatically. I like gitlab, in general.\nHowever, there\u0026rsquo;s one significant feature that Cloudflare Pages has, that Gitlab Pages does not:\nIPv6 Gitlab has had IPv6 as an open issue for years. I\u0026rsquo;ve actually been hopeful they\u0026rsquo;d resolve it, but alas, they don\u0026rsquo;t seem to be interested. But cloudflare can do it. It wasn\u0026rsquo;t even something I had to figure out how to turn on \u0026ndash; it works by default.\nCloudflare can link to your gitlab repo, and builds updates seemingly faster than I can alt-tab to my browser. There\u0026rsquo;s a ton of amazing features \u0026ndash; if you pay the big bucks. But even the free-tier account gives a decent selection of features: My site is now hosted worldwide in their CDN, rather than a single-core VM in Toronto. So next time somebody in Japan wants to figure out how to get PS2 Trackpoint scrolling to work in Wayland, they\u0026rsquo;ll have a decent page-load experience.\nSo that solves hosting the page content, but there\u0026rsquo;s still the matter DNS\u0026hellip; The problem is, I can\u0026rsquo;t change my A/AAAA records. Cloudflare\u0026rsquo;s global CDN means there\u0026rsquo;s a lot of fluctuating IP addresses that will serve my page. That\u0026rsquo;s fine, that\u0026rsquo;s what CNAMEs are for, right? Well, no. The URL I\u0026rsquo;ve always published is \u0026ldquo;chrisirwin.ca\u0026rdquo;, not \u0026ldquo;myblog.chrisirwin.ca\u0026rdquo;, etc. The boring bit is: You can\u0026rsquo;t do a top-level CNAME. Well, to be accurate, you can, but you can\u0026rsquo;t have any other subdomains. Since I use this domain for a bunch of other stuff still as well, CNAME on chrisirwin.ca is out of the question.\nI could move it to \u0026ldquo;blog.chrisirwin.ca\u0026rdquo; (and at time of writing, that\u0026rsquo;s a valid instance of this site, and where I was testing). However, I\u0026rsquo;d still need something listening on chrisirwin.ca and issuing 302 directives. Which means \u0026ldquo;Cloudflare pages and\u0026hellip;\u0026rdquo;\nIt turns out, cloudflare solved this, too\u0026hellip; IF you use them as your DNS provider. They\u0026rsquo;ll let you define a top-level CNAME, but knowing that causes problems, they\u0026rsquo;ll actually pro-actively resolve it at the DNS server-side before answering DNS requests.\nMy configuration says \u0026ldquo;chrisirwin.ca \u0026ndash;CNAME\u0026ndash;\u0026gt; cloudflare pages\u0026rdquo;, but a DNS query actually returns \u0026ldquo;chrisirwin.ca -A/AAAA-\u0026gt; ipv4/ipv6 address for cloudflare pages\u0026rdquo;.\nHandy.\nThen it was just migrating a DNSSEC-protected domain zone between providers. I spent a lot of time worried about this, reading docs and articles. It turns out, it works pretty much like the docs say: Disable DNSSEC, wait a day, update your nameserver records to point to cloudflare, then enable DNSSEC again (and copy cloudflare\u0026rsquo;s DNSSEC public key back to your registrar to add to your domain \u0026ndash; in my case, Gandi).\nSo now, a little over 24 hours since I started, this domain seems to be completely migrated. Unless you can\u0026rsquo;t read this post, then I didn\u0026rsquo;t configure something correctly, and you should disregard this article you didn\u0026rsquo;t just read.\nUpdate\n","permalink":"/posts/cloudflare-pages-hugo/","summary":"\u003cp\u003eThis is the first post after migrating this blog to cloudflare-pages hosted (pushed from gitlab.com). Why?\u003c/p\u003e","title":"Cloudflare Pages, Hugo, and You"},{"content":"A better, more complete (with dashboard + mobile automation sync) is available in a new article: Home Assistant Notifications (with Sync in Dashboard and Mobile)\nThe original article is preserved below. I encourage you to use the new one, though.\nI recently wanted to add Notifications to Home Assistant. Like anything in HA, there are multple methods to accomplish any task. Here\u0026rsquo;s how I used persistent notifications to accomplish this.\nHome Assistant has multiple methods of notifications, each with their own Pros and Cons:\nDevice Notifications - Works great for a phone, but on a wall-mounted tablet, you don\u0026rsquo;t see or check those notifications. They\u0026rsquo;re also not visible in HA dashboards.\nDiscord/etc - Mostly the same as above, except you get them on more than your phone.\nPersistent Notifications - This is a notification channel built-in to HA. Unfortunately, it\u0026rsquo;s not the most visible in either HA Dashboards (buried in the sidebar) or on your phone (no system notification).\nIf you use HACS in HA, you open the world to hundreds or thousands of additional cards for HA Dashboards. Unfortunately, there is only one to display persistent notifications: Home Feed Card. This doesn\u0026rsquo;t fit the eye-catching aesthetic I desire for a wall-mounted tablet.\nHowever, I did end up coming up with some basic widgets that require no addons (and some, below, with some minor, helpful addons):\nManual Entities If you have a pre-determined list of notification IDs, you can add them to an Entity List. This causes ugly errors if a notification doesn\u0026rsquo;t exist at the current time, however:\n- type: entities title: Manual Entities entities: - persistent_notification.abc - persistent_notification.xyz - persistent_notification.zzz If we instead switch to the built-in entity-filter card, you can have those errors obfuscated. Additionally, we\u0026rsquo;ll set \u0026lsquo;secondary_info\u0026rsquo; and use the attribute, so we can get more information about the notification:\n- type: entity-filter entities: - entity: persistent_notification.abc secondary_info: last-changed type: attribute attribute: message - entity: persistent_notification.xyz secondary_info: last-changed type: attribute attribute: message - entity: persistent_notification.zzz secondary_info: last-changed type: last-changed attribute: message state_filter: - operator: regex value: . card: type: entities title: Manual Filter Entities w/ Secondary That already looks a ton better. However, there are two issues still present:\nYou need to have a pre-determined list of notification IDs that will be displayed. Which means you need to update this card if you add new notifications.\nIt might not fit in with your card layout.\nAuto-Entities That\u0026rsquo;s where Auto-Entities comes in to play. It lets you do a few nice things. First, it lets us filter on entity_id as well as use wildcards, so we can match persistent_notification.*.\nThe first thing you should note here is the presence of my Washing Machine notification. It wasn\u0026rsquo;t listed in the previous examples, because I hadn\u0026rsquo;t hard-coded it. No need, here:\n- type: custom:auto-entities card: type: entities title: Notifications using Attribute \u0026amp; Date filter: include: - entity_id: persistent_notification.* options: type: attribute attribute: message secondary_info: last-changed The real magic is that you\u0026rsquo;re no longer bound to an entities card. You can use any container card (grid, vertical-stack, etc) as a container, and any card as content. Which lets us use more generic entity cards in a vertical stack. (Also note, they\u0026rsquo;re sorted by last_triggered):\n- type: custom:auto-entities card: type: vertical-stack title: Notifications in Vertical Stack of Cards card_param: cards filter: include: - entity_id: persistent_notification.* options: type: entity attribute: message sort: method: last_triggered Putting into use in a dashboard There\u0026rsquo;s one last trick I wanted to use them in a dashboard: show_empty: false, so the card is only visible when there are notifications.\nI\u0026rsquo;m additionally using two other addons from HACS:\nlayout-card to gridify my dashboard (the view_layout belongs to that). Canary, allowing some basic css styles (canary_style) This gives me very visible notifications when present:\nBut which disappear entirely when not set:\n- type: custom:auto-entities card: type: horizontal-stack view_layout: grid-area: h1 card_param: cards show_empty: false filter: include: - entity_id: persistent_notification.* options: type: entity attribute: message canary_style: border-color: lightcoral background-image: \u0026gt;- linear-gradient(rgba(255,0,0,0), rgba(240,128,128,0.3), rgba(240,128,128,0.8)); sort: method: last_triggered As for dismissing notifications, that wasn\u0026rsquo;t a goal. These notifications are cleared by automations when their status is corrected (i.e., opening the door for the Laundy, Locking the Front Door, etc).\nStill to come: I\u0026rsquo;m forwarding these notifications to my Android Phone as well, but still doing this with a pre-defined list of IDs. I\u0026rsquo;m still working on a generic forward Automation, and will post a follow-up with more when I\u0026rsquo;ve solved that.\n","permalink":"/posts/homeassistant-notifications-in-dashboard/","summary":"\u003cp\u003eA better, more complete (with dashboard + mobile automation sync) is available in a new article: \u003ca href=\"/posts/homeassistant-notifications-sync-dashboard-mobile\"\u003eHome Assistant Notifications (with Sync in Dashboard and Mobile)\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe original article is preserved below. I encourage you to use the new one, though.\u003c/p\u003e","title":"Home Assistant Notifications in Dashboard"},{"content":"I\u0026rsquo;ve recently been porting things from Samsung Smartthings to Home Assistant. Smartthings has a significantly lower learning curve, but suffered from a reliance on the cloud. While you have a local hub, any \u0026ldquo;custom\u0026rdquo; device handlers actually ran on the cloud on Samsung servers somewhere. Which means advanced functionality was often very, very slow.\nAs an example, I have a Schlage Z-Wave door lock, which worked out of the box with SmartThings, albeit with limited functionality. To resolve that issue, I used RBoyApps \u0026ldquo;Universal Enhanced Z-Wave Lock\u0026rdquo; Handler. It was a much, much improved interface for the lock, allowing configuring of advanced settings. Unfortunately, using that handler meant any device interaction ran a round trip through the Samsung cloud. Trying to unlock my door from my phone often took 10-15 seconds, which doesn\u0026rsquo;t sound like much, but is slower than just typing in my code manually.\nI finally decided to make the switch to Home Assistant \u0026ndash; a self-hosted, purely local platform.\nHA is much, much more hands-on. While you can do plenty of things in the UI, a lot of configuration requires manual adjustments to yaml config files. I don\u0026rsquo;t mind that. The wiki is very informative for most functionality one requires.\nHowever, some of the more advanced functionality is kind of written between the lines in the wiki. There\u0026rsquo;s references to templates, triggers (the template object specifically), and functions, but it doesn\u0026rsquo;t seem to have the same documentation coverage.\nMostly through trial and error, I\u0026rsquo;ve learned some functionality.\nTemplates You need to use \u0026gt;- to switch to templates instead of normal values. There is also \u0026gt;, and I\u0026rsquo;m not sure if the difference (if any)\n# Standard value key: value # Templated value key: \u0026gt;- {% if trigger %} Foo {% else %} Bar {% endif %} Functions have values piped into them like a unix command-line: {% value | function(args) }% Trigger information is available in templates There is some very, very basic coverage of this on the wiki, but it appears to be wrong/incomplete.\nYou can use a trigger object to see what triggered this function, as well as compare old/new states.\nHowever, the trigger object properties are different depending on the trigger. This makes sense, but is problematic. The trigger object (and it\u0026rsquo;s properties) differ if you\u0026rsquo;re triggering on the lock state, vs the lock entity.\n{{ trigger.to_state.name }} is {{ trigger.to_state.state }} The Door Lock Notification automation Below is a single automation that will notify all devices when the door is locked or unlocked. It can have multiple locks added as automation triggers, and it will use the trigger template object to output the correct notification data. It does handle a scenario of missing trigger data as well (not well, but well enough).\nIt also does a very basic status cleanup:\nReplacing the \u0026ldquo;by RF\u0026rdquo; note to specifically mention Home Assistant. This is any changes made via the web UI or apps. Replacing \u0026quot; by user 0\u0026quot; with nothing, because there is no user 0. This is typically locking the door via the keypad outside. We don\u0026rsquo;t need to identify this \u0026ldquo;user\u0026rdquo;. Replacing \u0026ldquo;user 1\u0026rdquo; and \u0026ldquo;user 2\u0026rdquo; with our names. I\u0026rsquo;d prefer this to be a proper lookup via !secret data. However, the exact usage of those is unclear as well, so that is an exercise for another day. Note that Home Assistant will unfortunately mangle this function into an unreadable mess if this is in your automations.yaml. Fixing that is an exercise to the reader\u0026hellip;\nautomation: - id: \u0026#39;1607152528\u0026#39; alias: Door Lock Notifications description: \u0026#39;\u0026#39; mode: parallel max: 10 trigger: - platform: state entity_id: lock.front_door action: - service: notify.notify data: title: \u0026gt;- {% if trigger %} {{ trigger.to_state.name }} is {{ trigger.to_state.state }} {% else %} Unkown Door - Status Changed {% endif %} message: \u0026gt;- {% if trigger %} {% set lock_status = trigger.to_state.attributes.lock_status %} {% set lock_status = lock_status|replace(\u0026#34;by RF\u0026#34;, \u0026#34;by Home Assistant\u0026#34;) %} {% set lock_status = lock_status|replace(\u0026#34; by user 0\u0026#34;, \u0026#34;\u0026#34;) %} {% set lock_status = lock_status|replace(\u0026#34;user 1\u0026#34;, \u0026#34;Chris\u0026#34;) %} {% set lock_status = lock_status|replace(\u0026#34;user 2\u0026#34;, \u0026#34;Alexis\u0026#34;) %} {{ lock_status }} {% else %} Missing trigger information {% endif %} ","permalink":"/posts/homeassistant-lock-notifications/","summary":"\u003cp\u003eI\u0026rsquo;ve recently been porting things from Samsung Smartthings to Home Assistant. Smartthings has a significantly lower learning curve, but suffered from a reliance on the cloud. While you have a local hub, any \u0026ldquo;custom\u0026rdquo; device handlers actually ran on the cloud on Samsung servers somewhere. Which means advanced functionality was often very, very slow.\u003c/p\u003e","title":"Home Assistant and Lock Notifications"},{"content":"A coworker graciously gave me an IBM RT3200! It\u0026rsquo;s a tenkeyless IBM keyboard with integrated trackpoint. However, I did have some issues getting scrolling working properly with a modern USB-based system running Wayland.\nPros, cons, and quirks It\u0026rsquo;s actually a pretty nice keyboard which has the following features:\nTrackpoint! Three mouse buttons (vs Two on the Unicomp EnduraPro, for example) Meta/Windows key (vs older Model M boards without it) Full format keyboard (vs compact or laptop-format with \u0026ldquo;thinkpad\u0026rdquo; keyboards) Tenkeyless (so no numpad) It does have some downsides:\nRubber dome (No clacky buckling springs) PS/2 ! No volume control (vs laptop-format thinkpad keyboards, which also have a Fn key) It also has the quirk of a PS/2 mouse passthrough, labelled \u0026ldquo;2 BUTTON MOUSE ONLY\u0026rdquo;. Never seen that before.\nUSB Adapter To use this on a modern machine necessitates two PS/2 to USB adapters, as the trackpoint and keyboard use separate plugs. I bought this dual adapter on Amazon, which seems to work fine. I\u0026rsquo;ll be honest, it\u0026rsquo;s best feature is it\u0026rsquo;s price. Curiously, it identifies as a Barcode reader, although it properly exposes multiple appropriate input devices:\n$ lsusb | grep 13ba Bus 001 Device 015: ID 13ba:0018 PCPlay Barcode PCP-BCG4209 Connecting this to any USB port gets the keyboard and pointer working mostly fine.\nScrolling On Linux, the trackpoint is supposed to scroll when the middle button is held down. It\u0026rsquo;s a very useful feature, and it did not work out of the box like it does with normal USB keyboards. This is because traditional USB devices have both a vendor and product ID, which can be used to identify it specifically.\nFor example, 17ef:6047 identifies my Lenovo (17ef) Thinkpad Compact USB Keyboard with Trackpoint (6047). This can then apply certain rules to the keyboard (keymappings, Fn key handling) and trackpoint (it knows it\u0026rsquo;s a trackpoint instead of a plain-old mouse or trackpad). On that device, middle-button scrolling works out of the box.\nThis keyboard has no unique identifiers (PS/2 predates that need). Furthermore, the PS/2 to USB adapter identifies itself as a barcode reader. I suspect this is simply due to cheap electronics re-using device IDs (whether surplus parts, re-used code, or just lack of care).\nFinding Device IDs There\u0026rsquo;s very little documentation on fixing this, and I only found the solution via a question on Ask Ubuntu, of all places. Luckily, it\u0026rsquo;s a trivial change consisting of a one-line udev rule to apply an environment variable to the device.\nYou\u0026rsquo;ll need the following information about your particular PS/2 adapter:\nUSB Vendor ID EVDev name USB Vendor ID can be found with lsusb. If you\u0026rsquo;re not sure which device is correct (i.e., it says \u0026ldquo;Barcode\u0026rdquo; instead of \u0026ldquo;PS/2 Keyboard\u0026rdquo;), you can check lsusb before and after plugging your device in, and find the new one. You can also check dmesg, which should list your new device.\nYou can use libinput to check for devices. There were 33 devices on my laptop due to the plethora of input devices I have connected (plus each device shows up multiple times). But I was able to pick out the new one becuse it said \u0026ldquo;Barcode\u0026rdquo;:\n$ sudo libinput list-devices | grep Device [...] Device: Barcode Reader Device: Barcode Reader Mouse Device: Barcode Reader System Control Device: Barcode Reader Consumer Control [...] Create udev rule: Armed with the above information, I can craft my udev rule. I\u0026rsquo;m matching based on the Vendor ID and the word \u0026ldquo;Mouse\u0026rdquo;:\n$ sudo tee /etc/udev/rules.d/70-ps2-pointingstick.rules \u0026lt;\u0026lt;EOF # Detect PS/2 Mouse as a pointingstick (Trackpoint) ACTION==\u0026quot;add|change\u0026quot;, ENV{ID_VENDOR_ID}==\u0026quot;13ba\u0026quot;, ATTRS{name}==\u0026quot;*Mouse*\u0026quot;, ENV{ID_INPUT_POINTINGSTICK}=\u0026quot;1\u0026quot; EOF After this, you can trigger udev to reload the rules, then unplug/replug your device.\n$ sudo udevadm control --reload-rules You should now have middle-mouse scrolling on your PS/2-to-USB-connected trackpoint.\n","permalink":"/posts/ps2-trackpoint-scrolling-wayland/","summary":"\u003cp\u003eA coworker graciously gave me an IBM RT3200! It\u0026rsquo;s a tenkeyless IBM keyboard with integrated trackpoint. However, I did have some issues getting scrolling working properly with a modern USB-based system running Wayland.\u003c/p\u003e","title":"PS/2 Trackpoint Scrolling in Wayland"},{"content":"Like a large number of people around the world right now, my place of work has closed the doors for safety concerns. I\u0026rsquo;m fortunate enough to be in a position to work from home via VPN, but I did want to keep some separation between my work and non-work environments.\nThe full-separation route would be creating a new user profile on my local machine. This would ensure complete separation. I didn\u0026rsquo;t go this route because I don\u0026rsquo;t need to keep everything separate. A fair amount of my work is already isolated via RDP (to a Windows machine) or ssh (to my workstation and development servers). The biggest remaining concern was Firefox \u0026ndash; keeping all my work logins (Office 365, issue tracker, etc) from intermingling with my personal accounts.\nThere are several ways to accomplish this:\nFirefox Containers is an easy one, is built-in, and easily configurable. Problem is it\u0026rsquo;s per-tab, so I can have a dozen tabs with different container contexts. Furthermore, I can\u0026rsquo;t just click the new-tab button and get to my issue tracker, I need to select the correct context every time.\nUse a different browser. I could install Chrome for work. I don\u0026rsquo;t want to.\nUse Firefox Profiles. This is the ideal solution for me, as it allows window-level separation, but also can provide alt-tab separation with an intelligent window manager (GNOME, for example).\nCreate a new profile $ firefox --no-remote -P There\u0026rsquo;s a few things going on here:\n--no-remote tells Firefox to start a new instance, instead of telling any running Firefox to start a new window\n-P will open the profile manager\nClick \u0026ldquo;Create Profile\u0026rdquo;, and give it a name. Simpler is better. I used \u0026ldquo;work\u0026rdquo;.\nCreate a desktop shortcut I need Gnome to know how to run and manage this Firefox instance.\nThe key details are that the name, class, and .desktop filename all match. This is the heuristic GNOME uses to group windows.\n$ cat .local/share/applications/firefox-work.desktop [Desktop Entry] Encoding=UTF-8 Name=Firefox for Work Comment=Firefox Work profile Type=Application Terminal=false Exec=firefox --name=firefox-work --class=firefox-work --no-remote -P work Icon=firefox # If you have a logo icon you want to use instead, you can use a path: #Icon=/path/to/work.png Note instead of just -P, we provided a profile name, -P work. This tells Firefox to run that profile, instead of the standard one.\nI strongly suggest setting a custom icon.\nRun Firefox for Work Now when you type \u0026ldquo;Firefox\u0026rdquo;, you\u0026rsquo;ll have an additional entry. Note that in this screenshot, I also have Firefox installed via flatpak, so I\u0026rsquo;ve got a third icon that can be safely ignored.\nThe nice thing is that this is grouped separately in ALT+TAB as well, so you can ALT+TAB to your work Firefox, hit CTRL+N, and bring up whatever logged-in work account you want. Your Work browsing is no longer intermingling with your personal browsing,\nFirefox always uses my work profile now! Run firefox -P again, select your normal default profile, and ensure the \u0026ldquo;use the selected profile without asking at startup\u0026rdquo; box is checked. This will re-set the default.\nPrivacy caveat Depending on your VPN, your personal Firefox may still be doing DNS lookups and possibly even routing via your work VPN, just like normal. This simply keeps profiles, accounts, passwords, and (most importantly) workflow separate.\n","permalink":"/posts/isolate-work-in-firefox/","summary":"\u003cp\u003eLike a large number of people around the world right now, my place of work has closed the doors for safety concerns. I\u0026rsquo;m fortunate enough to be in a position to work from home via VPN, but I did want to keep some separation between my work and non-work environments.\u003c/p\u003e\n\u003cp\u003eThe full-separation route would be creating a new user profile on my local machine. This would ensure complete separation. I didn\u0026rsquo;t go this route because I don\u0026rsquo;t need to keep \u003cem\u003eeverything\u003c/em\u003e separate. A fair amount of my work is already isolated via RDP (to a Windows machine) or ssh (to my workstation and development servers). The biggest remaining concern was Firefox \u0026ndash; keeping all my work logins (Office 365, issue tracker, etc) from intermingling with my personal accounts.\u003c/p\u003e","title":"Isolate Work in Firefox"},{"content":"During my most recent server-build, I had to tackle exposing host-managed files to VMs.\nPreviously, I used NFS. However, that was becoming increasingly troublesome, unfortunately due to IPv6. While I have an IPv6 connection and IPv6 network, my router will stop advertising the IPv6 network if the Internet goes down, causing connectivity issues. Additionally, the uncertainty of whether any connection will come as fe80:: vs my regular routable address was a pain (especially when my IPv6 network changed). Ideally, everything would use fe80:: internally, but it didn\u0026rsquo;t, and that was likely due to me mucking about with things.\nI eventually worked around the quirky issues by forcing IPv4 on the NFS clients.\nFast forward to today, and I decided to investigate using filesystem passthrough using virtio-9p. This is the \u0026ldquo;built-in\u0026rdquo; filesystem passthrough in libvirt+qemu.\nSimple? On the surface, it is actually very simple. You select a path, you give it a name, and you mount it in the client.\nIn practice, a lot more thought and configuration is required.\nOptions Driver There are several options for \u0026ldquo;Driver\u0026rdquo;:\nHandle Path Path is the option you want to use, here. I didn\u0026rsquo;t find a very good explanation of \u0026ldquo;Handle\u0026rdquo;.\nMode Mode is how UIDs and permissions are handled between guest and host. To quote the Qemu 9p documentation wiki:\nPassthrough\nFiles on the filesystem are directly created with client-user\u0026rsquo;s credentials.\nThis would be the simplest option if the VMs were run as root. However, Fedora runs the VMs as the qemu user (additionally, you can run VMs as your user). Effectively, this means that while the guest may try to set ownership permissions on a file, the host may block it based on the qemu\u0026rsquo;s user account.\nI tried to use this option, going so far as making my shared directory permissions an inadvisably-open 777.\nReading files works as expected, keeping their original UIDs and permissions, as long as you keep in mind that both the guest and host are applying permissions rules. Even root on the guest can\u0026rsquo;t read a file qemu on the host can\u0026rsquo;t access.\nWriting doesn\u0026rsquo;t really work, as the Qemu host process is running as a limited user account. So if you\u0026rsquo;re user chris in the VM, you can write a file to the share, but be unable to read it. The file was written to the host disk by the qemu-owned VM process, failed to set ownership to chris, leaving you with a file you created, but can not read. You\u0026rsquo;ll also get errors setting ownership/permissions.\nEvery file created will be owned by qemu (which may just be a non-descript UID on your guest). This is less than desirable.\nThis would be an option if you were running a VM as your user, and running apps in the VM as the same user.\nSquashed\nIt is equivalent to passthrough security model; the only exception is, failure of privileged operation like chown are ignored. This makes a passthrough like security model usable for people who run KVM as non root.\n(Note, the wiki is a little unclear and lists this as \u0026ldquo;none\u0026rdquo;. Pretty sure this is correct)\nAs described, this is effectively the same as Passthrough, but it pretends ownership and permissions changes worked (even though they didn\u0026rsquo;t).\nThis would be a slightly better option if you were running a VM as your user, and running apps in the VM as the same user.\nMapped\nFiles are created with qemu user credentials and the client-user\u0026rsquo;s credentials are saved in extended attributes.\nThis completely separates host and client permissions.\nOn the host, you would chown -R qemu:qemu all the files you intend to share with the client. (the qemu-owned process still needs permissions for these files)\nOn the guest, you can chown/chmod as much as you want. Any metadata changes made by the guest are stored in extended attributes. These are global (they\u0026rsquo;re not per-vm extended attributes), so if you share the same path with two guests, they\u0026rsquo;ll see the same UIDs.\nYou should confirm you\u0026rsquo;re backing up extended attributes.\nI used Mapped. It allows permissions on the host to be handled simply (just make qemu own everything), while permissions on the guest can also be set independently (for whatever user/service is using those files).\nSource Path This is the path on the host you want to export.\nex: /foo/bar\nTarget Path This is actually a label, not a path. Use it to identify this share to the guest.\nex: foobar\nExport filesystem as readonly mount This is handy if you want to ensure the guest can not write files back to the host. Particularly handy if you share the same directory to multiple machines, but want to ensure only one can modify the contents.\nConfigured Here\u0026rsquo;s a sample configuration that exports the path /foo/bar with the label foobar, using the mapped security model:\nMounting And this is how the guest would mount the filesystem:\nfoobar /mnt/foobar 9p trans=virtio,version=9p2000.L,_netdev 0 0 Testing Note: You may encounter problems here, and may need to skip ahead to the SELinux rules\nOn the guest, you can change the permissions and ownerships:\n$ chown root:root /mnt/foobar $ chmod 750 /mnt/foobar $ ls -ld /mnt/foobar drwxr-x---. 1 root root 96 Mar 3 23:31 /mnt/foobar Here\u0026rsquo;s what that looks like on the host. Note the ownership and permissions haven\u0026rsquo;t changed:\n$ ls -ld /foo/bar drwxr-xr-x. 1 qemu qemu 96 Mar 3 23:31 /foo/bar The guest values are stored as extended attributes:\n$ attr -l /foo/bar Attribute \u0026#34;virtfs.mode\u0026#34; has a 4 byte value for /foo/bar Attribute \u0026#34;virtfs.uid\u0026#34; has a 4 byte value for /foo/bar Attribute \u0026#34;selinux\u0026#34; has a 35 byte value for /foo/bar Attribute \u0026#34;virtfs.gid\u0026#34; has a 4 byte value for /foo/bar Note: I haven\u0026rsquo;t looked for an easy way to read these values. They\u0026rsquo;re binary, and dumping them doesn\u0026rsquo;t convert them to a printable character\nTimes are updated on the files as normal (mtime, etc).\nSELinux Guest On the guest, these filesystems actually get tagged as nfs_t, so all the standard SELinux bools to allow httpd/samba/etc to work with NFS work as expected.\n$ ls -ldZ /mnt/foobar drwxr-x---. root root system_u:object_r:nfs_t:s0 /mnt/foobar Host On the host, SELinux inserts itself into the permissions as well to restrict qemu to only properly tagged files.\nI used svirt_image_t (the same as the libvirt images directory). There may be a better tag, but I couldn\u0026rsquo;t immediately find one.\n$ semanage fcontext -a -t svirt_image_t \u0026#39;/foo/bar(/.*)?\u0026#39; $ restorecon -Rv /foo/bar $ ls -ldZ /foo/bar drwxr-xr-x. 1 qemu qemu system_u:object_r:svirt_image_t:s0 96 Mar 9 20:56 /foo/bar RHEL and CentOS Oh yeah, RHEL (and therefore CentOS) don\u0026rsquo;t enable the 9p filesystem driver in their kernel. You\u0026rsquo;re SOL if you want to stay on the stock kernel.\nHowever, there are two options to add 9p:\nKernel-plus. This is the stock kernel, but \u0026ldquo;plus\u0026rdquo; a bunch of features RHEL disabled. It is available in the centosplus repository, which is available (but disabled by default) on CentOS.\nInstall CentOS Plus kernel\nkernel-ml may also include support. I haven\u0026rsquo;t actually tested 9p on this, but btrfs did work (which it doesn\u0026rsquo;t on stock).\nElRepo kernel-ml\n","permalink":"/posts/sharing-host-files-with-kvm/","summary":"Adventures with sharing host files with a KVM VM using 9p and libvirt with SELinux.","title":"Sharing Host Files With KVM"},{"content":"While checking out some logs and google search analytics, I found that my post about Discard (TRIM) with KVM Virtual Machines has been referenced far more than I expected it to be. I decided to take this opportitnity to fact-check and correct that article.\nI migrated my server from a 10-year old Dual Xeon X5650 to a significantly more performant Ryzen 1600. I undertook the process of migrating and updating my VMs, ensuring the libvirt definitions were current, switching from NFS to 9p, etc.\nvirtio vs virtio-scsi My previous post made this observation:\nAll of my VMs were using virtio disks. However, they don’t pass discard through. However, the virtio-scsi controller does.\nIt appears that is no longer entirely true. At some point between October 2015 and March 2020 (when I\u0026rsquo;m writing this), standard virtio-blk devices gained discard support. Indeed, virtio-blk devices actually support discard out of the box, with no additional configuration required:\n[guest] $ sudo lsblk -o NAME,MOUNTPOINT,DISC-MAX,FSTYPE NAME MOUNTPOINT DISC-MAX FSTYPE vdb 2G └─vdb1 [SWAP] 2G swap vda 2G └─vda1 / 2G xfs My research shows this was qemu 4.0 that added this. Fedora 31 has qemu 4.1.\nIt is no longer required to convert to virtio-scsi.\nFWIW, I\u0026rsquo;m using machine type \u0026lsquo;pc-q35-4.1\u0026rsquo;. I expect this behaviour is only exposed on machines \u0026gt;= 4.0.\nDiscard vs unmap/shrink While the guest will support discards now, that doesn\u0026rsquo;t actually shrink the disks on the host. I haven\u0026rsquo;t performed any testing to confirm, but I suspect it is actually performing a passthrough to the underlying storage layer, allowing the SSD to do a proper Garbage Collection, without modifying the file on the host filesystem. That\u0026rsquo;s all well and good, and a perfectly sane default, but I do still want to shrink files on the host. These VMs are on a somewhat smaller drive, and I wish to reclaim space whenever possible.\nTo shrink the actual files on the host, the discard mode must be changed to \u0026lsquo;unmap\u0026rsquo;.\n\u0026lt;driver name=\u0026#39;qemu\u0026#39; type=\u0026#39;qcow2\u0026#39; discard=\u0026#39;unmap\u0026#39;/\u0026gt; This can be done with the same sed-magic used in my last post. Additionally, the option is actually exposed through virt-manager now:\nEnable fstrim As before, use fstrim to issue discards on-demand\n$ sudo fstrim -av Going forward, you can either add ‘discard’ to the mount options in fstab (which will probably hurt performance), or use fstrim periodically. I opted for fstrim, as it has a systemd timer unit that can be scheduled:\n$ sudo systemctl enable --now fstrim.timer ","permalink":"/posts/discard-with-kvm-2020/","summary":"\u003cp\u003eWhile checking out some logs and google search analytics, I found that my post about \u003ca href=\"/posts/discard-with-kvm/\"\u003eDiscard (TRIM) with KVM Virtual Machines\u003c/a\u003e has been referenced far more than I expected it to be. I decided to take this opportitnity to fact-check and correct that article.\u003c/p\u003e","title":"Discard (TRIM) with KVM Virtual Machines... in 2020!"},{"content":"Fedora\u0026rsquo;s installer will happily set up an encrypted install with root-on-lvm-on-luks (/boot is still unencrypted. Secure Boot might be handy here still). This is supported and works out of the box.\nHowever, while I\u0026rsquo;m present when I reboot this machine, it is also headless (no keyboard or monitor), so typing a passphrase at boot is problematic. But no problem, you can have up-to ten key slots for a LUKS partition, right? And you can use a keyfile for one of those slots, right? So it should just be a matter of updating the crypttab, right?\nWhile crypttab does allow you to set a keyfile, and it does indeed work for all other filesystems, it doesn\u0026rsquo;t work for /. It expects a (mounted) path to the keyfile, and the USB token won\u0026rsquo;t be mounted until after /. So there\u0026rsquo;s a bit of a chicken and egg situation: You can\u0026rsquo;t mount the keys to decrypt root until after root is decrypted.\nBut luckily we\u0026rsquo;re using systemd, and it actually does handle this scenario. It allows us to say which device contains the key, without having to expect it to be mounted. We just can\u0026rsquo;t use crypttab to do this. The documentation exists (and is generally well written), but it still took a lot of fiddling to get this to work correctly.\nUse man systemd-cryptsetup-generator for the details.\nNote: the crypttab(8) manpage appears to indicate crypttab supports this as well, but I just couldn\u0026rsquo;t get it to work. That said, I also had issues with a too-short timeout that caused a lot of headaches\u0026hellip;\nSet up Kernel Command Line Have keys working Before we begin, you should have your keys created and added to your LUKS partition. Create your key file the \u0026ldquo;normal\u0026rdquo; way (random data into text file with no newline at the end). There\u0026rsquo;s plenty of articles and howtos about this part, so I won\u0026rsquo;t get into this here.\nAdd parameters to grub\u0026rsquo;s configuration We\u0026rsquo;ll need to edit the GRUB_CMDLINE_LINUX in /etc/default/grub to add all the decryption instructions for systemd to process at boot time:\n$ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX GRUB_CMDLINE_LINUX=\u0026#34;resume=/dev/mapper/vgsystem-swap rd.luks.uuid=luks-926ee4d1-4cd8-43d7-97a3-07d41ed2a742 rd.lvm.lv=vgsystem/root rd.lvm.lv=vgsystem/swap rd.luks.key=926ee4d1-4cd8-43d7-97a3-07d41ed2a742=/my.luks.key:LABEL=keys rd.luks.options=926ee4d1-4cd8-43d7-97a3-07d41ed2a742=keyfile-timeout=30s rhgb quiet\u0026#34; Here\u0026rsquo;s the luks-specific details I needed to have:\nrd.luks.uuid rd.luks.uuid=luks-926ee4d1-4cd8-43d7-97a3-07d41ed2a742\nThis tells systemd to open this container. The format is luks + crypt uuid, and it seems to have special hardcoding for that format. It probably works with just the bare UUID. Alternatively, you can probably try using rd.luks.name to set a different name once it\u0026rsquo;s opened. This line was actually here from the installer, so I left it alone.\nYou can figure out your UUID from either the command line (it may already be here), /etc/crypttab, or cryptsetup luksDump /dev/sdXX | grep UUID\nrd.luks.key rd.luks.key=926ee4d1-4cd8-43d7-97a3-07d41ed2a742=/my.luks.key:LABEL=keys\nThis is the magic bit, which tells systemd where the key is. The key parts (aside from the UUID again):\n/my.luks.key\nThis is the path to the key file relative to the USB stick, not the full path when mounted. So /my.luks.key instead of /mnt/keys/my.luks.key.\nLABEL=keys\nThis is an identifier for the filesystem that contains the keys. I labelled the filesystem on my drive as \u0026ldquo;keys\u0026rdquo;.\nBe careful of the changing delimiters. It changes from = to : toward the end.\nrd.luks.options rd.luks.options=926ee4d1-4cd8-43d7-97a3-07d41ed2a742=keyfile-timeout=30s\nThis simply lengthens the timeout to probe the keyfile. This is fairly important, as the default timeout seems fairly short, and there\u0026rsquo;s a 75% chance that my USB device will not be probed yet. I really expected early boot to be far more deterministic, and this caused a lot of problems while troubleshooting. 30 seconds is far more time than needed (occasionally 10s was not enough).\nUpdate Grub: $ sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg\nRemove from crypttab At this point, you\u0026rsquo;d expect it to work, but it doesn\u0026rsquo;t. You\u0026rsquo;ll still get prompted for a passphrase. The manpage for systemd-cryptsetup-generator actually tells us why in the rd.luks.uuid description:\nIf /etc/crypttab contains entries with the same UUID, then the name, keyfile and options specified there will be used. Otherwise, the device will have the name \u0026ldquo;luks-UUID\u0026rdquo;.\nYou could also set rd.luks.crypttab=no on the command line, but I still want to use crypttab for my non-root filesystems. So comment out the entry for this device in /etc/crypttab, and rebuild your initrd\u0026hellip;\n$ sudo dracut -f\nNow you should be able to boot and have your disk unlocked automagically at boot.\nFuture investigation It would be nicer to have this configured in /etc/crypttab. Perhaps it does work, and it was merely my USB timeout issues that prevented it.\n","permalink":"/posts/fedora-encrypted-root-with-key/","summary":"\u003cp\u003eFedora\u0026rsquo;s installer will happily set up an encrypted install with root-on-lvm-on-luks (/boot is still unencrypted. Secure Boot might be handy here still). This is supported and works out of the box.\u003c/p\u003e\n\u003cp\u003eHowever, while I\u0026rsquo;m present when I reboot this machine, it is also headless (no keyboard or monitor), so typing a passphrase at boot is problematic. But no problem, you can have up-to ten key slots for a LUKS partition, right? And you can use a keyfile for one of those slots, right? So it should just be a matter of updating the crypttab, right?\u003c/p\u003e","title":"Fedora Encrypted Root With Automatic Key Unlocking"},{"content":"I recently decided to migrate this blog from IkiWiki to Hugo. I was expecting an ardous ordeal, but the entire task turned out to be fairly short painless.\nI\u0026rsquo;ve been somewhat disillusioned with posting to my (ikiwiki) blog lately:\nIt looks like web 1.0 (not that that is bad, but I\u0026rsquo;d like it to be snazzier) Google told me it\u0026rsquo;s demoting my site due to being mobile-hostile It\u0026rsquo;s using tools I don\u0026rsquo;t completely understand anymore (or really did in the first place) One of the git hooks wasn\u0026rsquo;t working somewhere As if to emphasize my procrastination in fixing those issues, I\u0026rsquo;ve actually written five blog posts since the last one I published over a year ago. I\u0026rsquo;ll be re-editing those for posting shortly.\nUltimately, I looked around and decided my needs would be better served by switching to Hugo.\nMuch easier theme support Documentation is decent While it fills a similar niche (markdown to static site generator), the tools are more comprehensible Mainly, Less dependent on git internals Live-view mode with a local server, so I can preview posts before I publish them. Additionally, there are a few other things I plan to support in the future:\nIntegration with gitlab\u0026rsquo;s CI infrastructure to auto-build pages when commits are pushed Hosting on gitlab pages Now, could I have hunkered down and figured out how to get ikiwiki working again? Sure. I could eventually get it to look pretty, too.\nHowever, I managed to migrate my entire site (which admittedly is only about ~50 posts) to Hugo in about 3 hours. Additionally, I was able to preserve all original permaurls for the articles. This was simplified by both supporting the same hierarchy I was using (\u0026quot;/posts/filename/\u0026quot;).\nAs both engines use the same markdown syntax, the actual migration of the posts was very minimally invasive, dealing mainly with the post metadata and inline images.\nI used the following script to perform the bulk convert:\n#!bin/bash for mdwn in *mdwn do echo \u0026#34;Converting $mdwn\u0026#34; base=$(basename $mdwn .mdwn) md=$base.md # Get the title, date, and tags in the new formats title=$(grep \u0026#39;\\[\\[\\!meta title\u0026#39; $mdwn | sed -e \u0026#39;s/\\[\\[!meta title=\\(.*\\)\\]\\]/title: \\1/;\u0026#39;) date=$(grep \u0026#39;\\[\\[\\!meta date\u0026#39; $mdwn | sed -e \u0026#39;s/\\[\\[!meta date=\\(.*\\)\\]\\]/date: \\1/;\u0026#39;) tags=$(grep \u0026#39;\\[\\[\\!tag\u0026#39; $mdwn | sed -e \u0026#39;s/\\[\\[!tag \\(.*\\)\\]\\]/\\1/; s/ /\u0026#34;, \u0026#34;/g; s/^/tags: [\u0026#34;/; s/$/\u0026#34;]/\u0026#39;) # If tags are blank, mark them as \u0026#34;untagged\u0026#34;. if [[ -z \u0026#34;$tags\u0026#34; ]]; then tags=\u0026#39;tags: [\u0026#34;untagged\u0026#34;]\u0026#39; fi # Insert the new tags into the new .md files echo -e \u0026#34;---\\n$title\\n$date\\n$tags\\n---\u0026#34; \u0026gt; $md # Strip the old tags from the mdwn sources, adding the result to the md files # Also rewrite images while we\u0026#39;re at it. sed -e \u0026#34;/^\\[\\[!meta title.*$/d\u0026#34; \\ -e \u0026#34;/^\\[\\[!meta date.*$/d\u0026#34; \\ -e \u0026#34;/^\\[\\[!meta guid.*$/d\u0026#34; \\ -e \u0026#34;/^\\[\\[!tag.*$/d\u0026#34; \\ -e s\u0026#39;/\\[\\[\\!img \\+\\(\\S*\\) .* alt=\\(\u0026#34;.*\u0026#34;\\)]]/![\\2](\\1)/g\u0026#39; \\ $mdwn \\ \u0026gt;\u0026gt; $md done References:\nKai Hendry\u0026rsquo;s \u0026ldquo;static site migration from ikiwiki to hugo\u0026rdquo; ","permalink":"/posts/migrate-ikiwiki-to-hugo/","summary":"\u003cp\u003eI recently decided to migrate this blog from IkiWiki to Hugo. I was expecting an ardous ordeal, but the entire task turned out to be fairly short painless.\u003c/p\u003e","title":"Migrate From Ikiwiki to Hugo"},{"content":"Getting updated Windows media is certainly easier than it used to be. Back in the day, you\u0026rsquo;d have to slipstream updates into your installation media. It was a pain, and prone to cause problems (mainly because you could slipstream other things in there, too).\nMicrosoft has seen the light and provides ISO downloads of the current version of Windows (Note: This is not a \u0026ldquo;free\u0026rdquo; Windows license \u0026ndash; you still need to pay for that, or install on a machine which has been previously licensed).\nActually using that ISO is not quite straight forward. You could burn it to a double-sided DVD, but I don\u0026rsquo;t even own a computer with an optical drive currently. USB is the way to go.\nWith years of experience creating bootable linux USBs, I did the standard thing:\n$ dd if=Win10_1809Oct_English_x64.iso of=/dev/sdX However, while Linux ISOs are typically multi-boot images (files that can boot both CD and USB, on both BIOS and UEFI machines), it appears Microsoft has not leveraged that knowledge. This USB fails to boot.\nThe only official way to create USB-bootable Windows media is to use the \u0026ldquo;Media Creation Tool\u0026rdquo; on Windows. You may encounter the same issue I did: I don\u0026rsquo;t have Windows installed yet!\nThere is an alternate way, using a project called woeusb.\nWoeUSB Running the GUI simply doesn\u0026rsquo;t work, as more command-line options appear to be required.\nOne of the files in the image is larger than 4GB, which requires using NTFS on the USB disk. The following command is what I used to flash the Windows 10 1809 ISO to my USB stick (be sure to get the correct device name for your USB \u0026ndash; this command will wipe it).\n$ sudo woeusb --target-filesystem NTFS --verbose --device Win10_1809Oct_English_x64.iso /dev/sdX Be aware that this USB media will not boot with secure-boot enabled, though the resulting Windows 10 installation will work with secure-boot.\n","permalink":"/posts/windows10-iso-from-linux/","summary":"\u003cp\u003eGetting updated Windows media is certainly easier than it used to be. Back in the day, you\u0026rsquo;d have to slipstream updates into your installation media. It was a pain, and prone to cause problems (mainly because you could slipstream \u003cem\u003eother\u003c/em\u003e things in there, too).\u003c/p\u003e\n\u003cp\u003eMicrosoft has seen the light and provides \u003ca href=\"https://www.microsoft.com/en-ca/software-download/windows10ISO\"\u003eISO downloads\u003c/a\u003e of the current version of Windows (Note: This is not a \u0026ldquo;free\u0026rdquo; Windows license \u0026ndash; you still need to pay for that, or install on a machine which has been previously licensed).\u003c/p\u003e","title":"Windows 10 Bootable USB from ISO on Linux"},{"content":"I\u0026rsquo;ve spent the better part of one and a half years with the DDPai M6 Plus. I bought it based on the positive Techmoan review.\ntl;dr: I\u0026rsquo;m somewhat disappointed in the device. Something simpler like an A119S is a much better buy.\nSome background on me and dashcams I had a Mini 0805 in my last car, and have two Mini 0805\u0026rsquo;s (front+rear) in my second car (I just had the two cameras, and moved the front one into whichever vehicle I was driving). I was actually quite happy with the cameras, despite some complaints online (overheat, soft/blurry video, etc).\nHowever, in the spring I noticed the audio was intermittantly poor on one of the cameras. Sometimes it would work, sometimes it wouldn\u0026rsquo;t. I decided to live with it. However, I recently purchased a new car, and decided to install a new, working camera. I decided I wanted some new features the 0805 didn\u0026rsquo;t offer:\nParking mode Wifi to retrieve an \u0026ldquo;occurrence\u0026rdquo; from my phone While doing my research, Techmoan reviewed the DDPai M6 plus. He seemed quite impressed, and it seemed to check off all my boxes. I bought one, and hard-wired it in.\nOn with the review I purchased the camera in July 2016, and I\u0026rsquo;ve written this over the following 1.5 years (failing to post it sooner, it has been revised accordingly).\nGood Parking mode works\u0026hellip; Remote trigger, wifi + app, easily removable. Basically everything Techmoan said. Fits behind mirror (just barely, though). Excellent build quality. Feels very solid. Bad Without additional hardware, parking mode will kill your battery. LiIon battery instead of a capacitor. Big. This camera just barely fits behind my mirror. It is much more noticable than my 0805 ever was. Wifi is always on, so my phone tries to connect to it periodically. Be sure to change the passphrase. The app isn\u0026rsquo;t really that good. Took me a while to find how to download regular non-saved video (even after looking it up), and found I can only do it in smaller clips at a atime. Effectively, you need to yank the SD card to retrieve large amounts of video. This isn\u0026rsquo;t the end of the world (it\u0026rsquo;s faster), but means the wifi feature is effectively not used. The app has cloud features that don\u0026rsquo;t work when connected to the camera due to the lack of an internet connection when connected to camera\u0026rsquo;s wifi. It speaks Chinese when it turns on. 25fps. Weird. Although I guess for the purpose, this doesn\u0026rsquo;t really matter. My biggest problem has been with the parking mode. It will frequently simply record realtime video for hours on end, instead of entering the timelapse mode. This causes unecessary writes to the SD Card, and overwrites previous clips much earlier than expected. Furthermore, it will drain my car battery in two or three days without using my car. In colder months, this has lead to a significant use of my booster pack on Monday mornings. There is additional hardware that can solve this, but factor that into the purchase price.\nI also found the wifi itself very hard to use, since my phone didn\u0026rsquo;t want to stay connected to it due to the lack of an internet connection. I can\u0026rsquo;t fault DDPAI for this, as Android is just trying to make sure I get my email. A \u0026ldquo;Use only for local network\u0026rdquo; checkbox in Android\u0026rsquo;s wifi settings would be helpful.\nAdditionally, the wifi is very slow, short range, and prone to disconnects. I\u0026rsquo;ve taken to simply yanking the card to retrieve video, but since it has no screen, the app is still required for various functionality (periodic SD formatting, configuration, etc). A camera with a built-in screen is much easier to deal with.\nUgly It gets very hot during the day Particularly after recording for a while, the sound can be noticably delayed from the video. See this sample of cars driving past me. Also, at 1:00 I drive over some tracks, and the sound is delayed by about a second. (although this was solved with a firmware update. However, there are more recent firmware updates that I can\u0026rsquo;t apply due to the app simply failing to push the firmware over). tl;dr It\u0026rsquo;s not necessarily a bad device, it\u0026rsquo;s just not a great one either. $173 CDN is quite a steep price for a camera with a LiIon battery and functionality issues. Ultimately, a capacitor version with battery-drain protection would go a long way toward convincing me the higher price tag is worth it, even if the app stayed as-is.\nUltimately, it\u0026rsquo;s better than my 0805, but not enough to justify the $80 extra. The A118-C was $75 cheaper than the M6 Plus (and the A119S is available at that price now). you\u0026rsquo;d lose parking (although that might not be wisely usable on the M6 Plus) and wifi (although it\u0026rsquo;s not really a great feature), but you gain a capacitor, a much slimmer design, and a truely stand-alone device.\ntl;dr the tl;dr: Probably buy something else\n","permalink":"/posts/m6plus/","summary":"\u003cp\u003eI\u0026rsquo;ve spent the better part of one and a half years with the DDPai M6 Plus. I bought it based on the \u003ca href=\"http://www.techmoan.com/blog/2016/6/23/dashcam-review-ddpai-m6-is-the-best-mini-cam-ive-used.html\"\u003epositive Techmoan review\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003etl;dr: I\u0026rsquo;m somewhat disappointed in the device. Something simpler like an A119S is a much better buy.\u003c/p\u003e\n\u003ch1 id=\"some-background-on-me-and-dashcams\"\u003eSome background on me and dashcams\u003c/h1\u003e\n\u003cp\u003eI had a Mini 0805 in my last car, and have two Mini 0805\u0026rsquo;s (front+rear) in my second car (I just had the two cameras, and moved the front one into whichever vehicle I was driving). I was actually quite happy with the cameras, despite some complaints online (overheat, soft/blurry video, etc).\u003c/p\u003e","title":"DDPai M6 Plus Dashcam Review"},{"content":"If you\u0026rsquo;re considering trying out autocross, I say go for it. I\u0026rsquo;m very new, and have found people at the two events I\u0026rsquo;ve attended (WOSCA #1, and PITL #3) to be friendly and extremely helpful.\nAdditionally, you can do it with your own car. You don\u0026rsquo;t need some sort of special race-spec track beast. Both events have also had loaner helmets available (although I spent $200 and bought my own helmet meeting the appropriate standards).\nI\u0026rsquo;ve posted a video of the PITL #3 event on youtube. Additionally, I\u0026rsquo;ve created a playlist, which also contains my WOSCA videos.\nVideo! When I left off after my third adventure with ffmpeg, I decided that I wanted to attempt video transitions next time. Well, it\u0026rsquo;s next time.\nIndividual Clip Preparation Like always, I concatenated my dashcam video chunks, then trimmed them as appropriate. This was done like my previous posts, using -codec copy to preserve quality.\nTimelapse I decided to attempt a timelapse intro/outro of my drive to the event. 32 times seemed to be a good balance of not-too-long, while also being somewhat fluid.\nffmpeg -i 0-commute.mov -filter:v \u0026quot;setpts=0.03125*PTS\u0026quot; -an -strict -2 0-commute-timex32.mov; That command is the trimmed commute clip as input. The filter is basically speeding up the playback. 1*PTS would be normal speed, and 1/32 = 0.03125.\nI did not timelapse the audio, although that is possible too.\nOther prep I also made title images in Gimp, like before.\nPerformance considerations I went through several iterations to reach the final video I wanted. The best resource I had was this post by Mulvya on Stack Overflow.\nInitially, I had five video streams, configured to fade in/out as required. However, I very quickly hit a performance issue related to ffmpeg\u0026rsquo;s overlay filter. Previously I used concat, but performing a transition requires the videos to actually overlay.\nTo illustrate the issue, assume you have two one-minute videos, and you want a two-second transition between them. Your final video will be 1:98. ffmpeg will stretch the 1 minute video\u0026rsquo;s last frame to the length of the video, which means it is now compositing 98 additional seconds that could technically be preserved unchanged. Attempting to work around this (by fading out the source video, trimming it, etc) seem to basically mean you\u0026rsquo;re compositing a transparency. The performance is still affected.\nThen remember I\u0026rsquo;m compositing five videos together, each one building on the previous. This very quickly went from real-time to 5fps processing time.\nThe \u0026ldquo;solution\u0026rdquo; is to chunk each clip into three parts: A A short beginning and ending clip a few seconds long, and a middle clip that requires no modifications. Now you can overlay the two 2 second clips to create your transition, and concat the other 1:96.\nThe performance benefit is substantial.\nThe Video Build Command The command I ended up with is basically a big mess, so I\u0026rsquo;ll just get start with it, then pick apart each section:\n4 time ffmpeg \\ 5 -i 0-commute-timex32.mov \\ 6 -i 1-Run1-trim.mov \\ 7 -i 2-Run2-trim.mov \\ 8 -i 3-Run3-trim.mov \\ 9 -i 4-Run4-trim.mov \\ 10 -i 5-commute-timex32.mov \\ 11 -loop 1 -i ../../../PITL_logo.png \\ 12 -loop 1 -i ../../../0-PITL.png \\ 13 -loop 1 -i ../../../1-Run1.png \\ 14 -loop 1 -i ../../../2-Run2.png \\ 15 -loop 1 -i ../../../3-Run3.png \\ 16 -loop 1 -i ../../../4-Run4.png \\ 17 -loop 1 -i ../../../5-Run5.png \\ 18 -filter_complex \u0026quot; 19 [0:v] trim=start=0:end=15,setpts=PTS-STARTPTS [clip0start]; 20 [0:v] trim=start=15:end=170,setpts=PTS-STARTPTS [clip0]; 21 [0:v] trim=start=170:end=172,fade=out:st=170:d=1:alpha=1,setpts=PTS-STARTPTS [clip0end]; 22 23 [1:v] trim=start=0:end=10,setpts=PTS-STARTPTS [clip1start]; 24 [1:v] trim=start=10:end=88,setpts=PTS-STARTPTS [clip1]; 25 [1:v] trim=start=88:end=90,fade=out:st=88:d=1:alpha=1,setpts=PTS-STARTPTS [clip1end]; 26 27 [2:v] trim=start=0:end=10,setpts=PTS-STARTPTS [clip2start]; 28 [2:v] trim=start=10:end=106,setpts=PTS-STARTPTS [clip2]; 29 [2:v] trim=start=106:end=108,fade=out:st=106:d=1:alpha=1,setpts=PTS-STARTPTS [clip2end]; 30 31 [3:v] trim=start=0:end=10,setpts=PTS-STARTPTS [clip3start]; 32 [3:v] trim=start=10:end=106,setpts=PTS-STARTPTS [clip3]; 33 [3:v] trim=start=106:end=108,fade=out:st=106:d=1:alpha=1,setpts=PTS-STARTPTS [clip3end]; 34 35 [4:v] trim=start=0:end=10,setpts=PTS-STARTPTS [clip4start]; 36 [4:v] trim=start=10:end=98,setpts=PTS-STARTPTS [clip4]; 37 [4:v] trim=start=98:end=100,fade=out:st=98:d=1:alpha=1,setpts=PTS-STARTPTS [clip4end]; 38 39 [5:v] trim=start=0:end=10,setpts=PTS-STARTPTS [clip5start]; 40 [5:v] trim=start=10,setpts=PTS-STARTPTS [clip5]; 41 42 [6:v] trim=start=0:end=15,fade=out:st=3:d=1:alpha=1,setpts=PTS-STARTPTS [logo]; 43 [7:v] trim=start=0:end=15,fade=in:st=5:d=1:alpha=1,fade=out:st=12:d=1:alpha=1,setpts=PTS-STARTPTS [title0]; 44 [8:v] trim=start=0:end=9,fade=in:st=3:d=1:alpha=1,fade=out:st=7:d=1:alpha=1,setpts=PTS-STARTPTS [title1]; 45 [9:v] trim=start=0:end=9,fade=in:st=3:d=1:alpha=1,fade=out:st=7:d=1:alpha=1,setpts=PTS-STARTPTS [title2]; 46 [10:v] trim=start=0:end=9,fade=in:st=3:d=1:alpha=1,fade=out:st=7:d=1:alpha=1,setpts=PTS-STARTPTS [title3]; 47 [11:v] trim=start=0:end=9,fade=in:st=3:d=1:alpha=1,fade=out:st=7:d=1:alpha=1,setpts=PTS-STARTPTS [title4]; 48 [12:v] trim=start=0:end=9,fade=in:st=3:d=1:alpha=1,fade=out:st=7:d=1:alpha=1,setpts=PTS-STARTPTS [title5]; 49 50 [clip0start][logo] overlay [clip0logo]; 51 [clip0logo] [title0] overlay [clip0transition]; 52 53 [clip1start][title1] overlay [clip1title]; 54 [clip1title][clip0end] overlay [clip1transition]; 55 56 [clip2start][title2] overlay [clip2title]; 57 [clip2title][clip1end] overlay [clip2transition]; 58 59 [clip3start][title3] overlay [clip3title]; 60 [clip3title][clip2end] overlay [clip3transition]; 61 62 [clip4start][title4] overlay [clip4title]; 63 [clip4title][clip3end] overlay [clip4transition]; 64 65 [clip5start][title5] overlay [clip5title]; 66 [clip5title][clip4end] overlay [clip5transition]; 67 68 [clip0transition] [clip0] 69 [clip1transition] [clip1] 70 [clip2transition] [clip2] 71 [clip3transition] [clip3] 72 [clip4transition] [clip4] 73 [clip5transition] [clip5] concat=n=12 [vout] 74 \u0026quot; \\ 75 -map \u0026quot;[vout]\u0026quot; \\ 76 -aspect '16:9' \\ 77 -codec:v libx264 -crf 21 -bf 2 -flags +cgop -pix_fmt yuv420p \\ 78 -movflags faststart \\ 79 -strict -2 \\ 80 00_final_video.mp4 Yeah\u0026hellip;\nLines 5-17 are my source input files. All visual files are here (video, overlays, etc).\nLines 19-21 are the chunking of the first video into start/main/end segments. This is then repeated for each clip through to Line 40\nsetpts to guarantee every video segment starts its counters at 0 (instead of their pre-existing timestamps). trim is used to cut the video. Values calculated by hand. The end of each video has a fade-out. You could do a fade-in on the start segment, but that has additional quirks I\u0026rsquo;ll discuss shortly. Lines 42-48 are the title cards (png files from Gimp) Each configured with a fade-in, and fade-out, and trim values. Line 53/54 are the short overlay transitions. Line 53 overlays the title card on the starting video Line 54 overlays the fade-out transition of the ending video on-top of the output from Line 53 It is important to note here that you could use a fade-in on the start video instead. However, you\u0026rsquo;ll need your fade-out to be the exact correct length, otherwise your fade-in will be truncated, and/or your fade-in won\u0026rsquo;t transition seamlessly to the rest of the clip. This way was easier.\nLines 50/51 are very similar, but for the Logo and main title.\nLines 68-73 simply concatenate the short transition overlays we\u0026rsquo;ve created, with the untouched video clips inbetween.\nThis saves soooo much processing time.\nLine 75 maps the final concat output to the file\nLine 76 fixes the aspect ratio\nLine 77-79 sets the youtube video codec stuff\nLine 80 is the output file name.\nNow, that\u0026rsquo;s pretty complex.\nThe Audio Build Command You also might notice there\u0026rsquo;s no sound. Attempting to do sound at the same time caused a number of buffer issues, video stutter, sound not lining up. Furthermore, while this video renders \u0026ldquo;faster\u0026rdquo; than a pure-overlay approach, we\u0026rsquo;re still talking \u0026gt;10 minutes, so trial-and-error to fix sound was very time consuming. I eventually decided to just do the sound separately, then merge them afterwards.\n100 time ffmpeg \\ 101 -itsoffset 0 -i ../../../Alternate.mp3 \\ 102 -i 1-Run1-trim.mov \\ 103 -i 2-Run2-trim.mov \\ 104 -i 3-Run3-trim.mov \\ 105 -i 4-Run4-trim.mov \\ 106 -itsoffset 0 -i ../../../Drifting_2.mp3 \\ 107 -filter_complex \u0026quot; 108 [0:a] atrim=start=0:end=171,asetpts=PTS-STARTPTS [clip0sound]; 109 [1:a] atrim=start=0:end=89,asetpts=PTS-STARTPTS [clip1sound]; 110 [2:a] atrim=start=0:end=107,asetpts=PTS-STARTPTS [clip2sound]; 111 [3:a] atrim=start=0:end=107,asetpts=PTS-STARTPTS [clip3sound]; 112 [4:a] atrim=start=0:end=99,asetpts=PTS-STARTPTS [clip4sound]; 113 [5:a] atrim=start=0:end=246,asetpts=PTS-STARTPTS [clip5sound]; 114 115 [clip0sound] [clip1sound] acrossfade=d=1 [a01]; 116 [a01] [clip2sound] acrossfade=d=1 [a012]; 117 [a012] [clip3sound] acrossfade=d=1 [a0123]; 118 [a0123] [clip4sound] acrossfade=d=1 [a01234]; 119 [a01234] [clip5sound] acrossfade=d=1 [aout] 120 121 \u0026quot; \\ 122 -map \u0026quot;[aout]\u0026quot; \\ 123 -vn \\ 124 -codec:a aac -strict -2 -b:a 384k -r:a 48000 \\ 125 -movflags faststart \\ 126 -strict -2 \\ 127 00_final_audio.mp4 This command is actually very similar. Interestingly, there is a crossfade functionality for audio. Why this doesn\u0026rsquo;t exist for video, I\u0026rsquo;ll never know.\nLines 101-106 are the input files. Note that I used mp3 files for the timelapses. These are from the Youtube Audio Library (attributed appropriately on the youtube video)\nLines 108-113 are similar time clipping, using values similar to the source videos (but all 1 second off, due to the two-second overlap I used for the videos)\nLines 115-119 are stringing the audio together using the acrossfade filter.\nMerging Audio and Video ffmpeg -i 00_final_video.mp4 -i 00_final_audio.mp4 -codec copy 00_final_merged.mp4 Not much too it. You may need to do a few rounds of Audio \u0026amp; Merging to ensure the sound lines up with the video.\n","permalink":"/posts/pitl-17-3/","summary":"\u003cp\u003eIf you\u0026rsquo;re considering trying out autocross, I say go for it. I\u0026rsquo;m very new, and have found people at the two events I\u0026rsquo;ve attended (WOSCA #1, and PITL #3) to be friendly and extremely helpful.\u003c/p\u003e\n\u003cp\u003eAdditionally, you can do it with your own car. You don\u0026rsquo;t need some sort of special race-spec track beast. Both events have also had loaner helmets available (although I spent $200 and bought my own helmet meeting the appropriate standards).\u003c/p\u003e","title":"Push It To the Limit #3"},{"content":"Goal I want to combine a few clips together, with a 5-seconds of intro text on each one.\nCreate overlay text in GIMP I created some overlay text in gimp, then exported to png files. An example (Note the transparancy, and drop shadow):\nTrim clips to length Using the methods I\u0026rsquo;ve described in previous ffmpeg posts, I trimmed the clips, ensuring that there is at least five seconds of lead-in on each clip for the text.\nffmpeg -i 1-Tire-Squeal-Front.MOV -ss 1:01 -to 1:24 -c copy 1-Tire-Squeal-Front.trim.MOV Overlay text on video clips Then I overlayed the PNG on top of the video for five seconds, thanks to Google leading me to mark4o on Super User:\nffmpeg -i 1-Tire-Squeal-Front.trim.MOV -loop 1 -i 1-Tire-Squeal-Front.png -filter_complex \u0026quot;[1:v] fade=out:st=5:d=1:alpha=1 [ov]; [0:v][ov] overlay=0:0 [v]\u0026quot; -map \u0026quot;[v]\u0026quot; -map 0:a -codec:v libx264 -crf 21 -bf 2 -flags +cgop -pix_fmt yuv420p -codec:a aac -strict -2 -b:a 384k -r:a 48000 -movflags faststart -to 0:23 1-Tire-Squeal-Front.title.mp4 I basically just used his example, except I changed the duration, and offset since my title png is already 1920x1080. Also, all that youtube codec stuff.\nI did have trouble with the last few seconds of video being clipped with -shortest. I ended up specifying the appropriate -to length.\nOverlay trailing text on video clip I wanted an ending text on the video, but was having trouble getting two clips to match exactly (and doing the video like the above). Then I realized I could just use a slightly different filter:\nffmpeg -i 3-Bob.trim.MOV -loop 1 -i 3-Bob.png -loop 1 -i 4-Oh.png -filter_complex \u0026quot;[1:v] fade=out:st=5:d=1:alpha=1 [ov]; [2:v] fade=in:st=16:d=1:alpha=1 [oe]; [0:v][ov] overlay=0:0 [v]; [v][oe] overlay=0:0 [vf]\u0026quot; -map \u0026quot;[vf]\u0026quot; -map 0:a -codec:v libx264 -crf 21 -bf 2 -flags +cgop -pix_fmt yuv420p -codec:a aac -strict -2 -b:a 384k -r:a 48000 -movflags faststart -to 21 4-Bob.title.mp4 This creates a third input stream, which is also a looped image. We create stream [oe] (overlay end), which does a fade-in at 16 seconds. We then take [v] and overlay [oe] on it, creating [vf]. We then map [vf] to our output.\nDone The video is what I set out to make. It might have been nicer to do some audio fades, but I\u0026rsquo;ll live.\nNext time Now that I\u0026rsquo;m done, I think it would have been a better idea to have the overlays fade in and out, as well as figure out better scene transitions. Next time\u0026hellip;\n","permalink":"/posts/highlight-reel/","summary":"\u003ch1 id=\"goal\"\u003eGoal\u003c/h1\u003e\n\u003cp\u003eI want to combine a few clips together, with a 5-seconds of intro text on each one.\u003c/p\u003e\n\u003ch1 id=\"create-overlay-text-in-gimp\"\u003eCreate overlay text in GIMP\u003c/h1\u003e\n\u003cp\u003eI created \u003ca href=\"overlay.xcf\"\u003esome overlay text in gimp\u003c/a\u003e, then exported to png files. An example (Note the transparancy, and drop shadow):\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"\u0026ldquo;sample overlay png\u0026rdquo;\" loading=\"lazy\" src=\"/posts/highlight-reel/overlay.png\"\u003e\u003c/p\u003e\n\u003ch1 id=\"trim-clips-to-length\"\u003eTrim clips to length\u003c/h1\u003e\n\u003cp\u003eUsing the methods I\u0026rsquo;ve described in previous ffmpeg posts, I trimmed the clips, ensuring that there is at least five seconds of lead-in on each clip for the text.\u003c/p\u003e","title":"Highlight Reel"},{"content":"While driving through New York on my way to the Watkins Glen 2017 opening weekend, I encountered a nut in a Volvo zig-zagging through highway traffic. I decided use ffmpeg yet again (I\u0026rsquo;m starting to wonder if OpenShot would actually bring anything to the table at this point, besides crashing)\nPlan I want a video with the rear camera until the Volvo passes, then the front camera after. I want to use the front audio for the whole video.\nSync Luckily, the front \u0026amp; rear videos were synced to within a fraction of a second. Close enough.\nTrim Rear video, starting at 2:40, ending at 4:24, when the car passes me.\nffmpeg -i ../Speedy_Gonzales-Rear.MOV -ss 2:40 -to 4:24 -c copy Rear.trim.MOV Front video, starting at 2:40, but ending at 4:57, when I want the video to end\nffmpeg -i ../Speedy_Gonzales-Front.MOV -ss 2:40 -to 4:57 -c copy Front.trim.MOV Combine This was pretty easy. I used an overlay as usual, but didn\u0026rsquo;t perform the scaling. I also instructed the overlay filter to continue with the original video when complete:\ntime ffmpeg -i Front.trim.1080.MOV -i Rear.trim.MOV -filter_complex \u0026quot;overlay=0:0:pass\u0026quot; -codec:v libx264 -crf 21 -bf 2 -flags +cgop -pix_fmt yuv420p -codec:a aac -strict -2 -b:a 384k -r:a 48000 -movflags faststart Combined-FrontAudio.mp4 You\u0026rsquo;ll note I\u0026rsquo;m using the same ffmpeg voodoo I used last time to help youtube processing time.\nI was expecting to have to remove the rear audio, but interestingly enough, it was gone already. I suspect the overlay filter killed it? No idea, actually.\nCrop Except not so fast! I had acidentally recorded both videos at different resolutions. The rear was doing 1920x1080, while the front was 2560x1080. This causes weird video like this:\nAdditionally, while 2560x1080 looks great to me (I have an ultrawide monitor), it letterboxes on a standard 1920x1080 display. Additionally, youtube adds that letterboxing during their processing, so watching my own 2560 video on youtube gets black bars on all four sides. Lovely. So I need to crop the video as well:\ntime ffmpeg -i Front.trim.MOV -i Rear.trim.MOV -filter_complex \u0026quot;overlay=0:0:pass,crop=1920:1080:0:0\u0026quot; -codec:v libx264 -crf 21 -bf 2 -flags +cgop -pix_fmt yuv420p -codec:a aac -strict -2 -b:a 384k -r:a 48000 -movflags faststart -to 10 Combined-FrontAudio.mp4 I\u0026rsquo;m fine with keeping the left of the front frame, as it looks \u0026ldquo;good enough\u0026rdquo;. If I wasn\u0026rsquo;t, I\u0026rsquo;d have had to have adjusted the overlay offset, as well as the crop offset.\nJust as a note, I fixed the resolution (and removed the datestamp from the rear camera) before I hit the track on Saturday. So I didn\u0026rsquo;t have to the cropping for the videos I put on youtube earlier (but filmed chronologically after).\nCPU takes it easy My CPU seems to be taking it easy today. Maybe its upset I made it work on Easter? Or this task just didn\u0026rsquo;t parallelize as well.\nDone The finished video turned out as well as I had hoped. I left in the first 90 seconds (despite mostly being a Pringles-related discussion) because, like any traffic video online, there often isn\u0026rsquo;t enough context. Also, it\u0026rsquo;s always the cammer\u0026rsquo;s fault, so I guess more incriminating evidence\u0026hellip;\n","permalink":"/posts/zigzag-volvo/","summary":"\u003cp\u003eWhile driving through New York on my way to the Watkins Glen 2017 opening weekend, I encountered a \u003ca href=\"https://youtu.be/8KceTjkQdvw?t=90\"\u003enut in a Volvo zig-zagging through highway traffic\u003c/a\u003e. I decided use ffmpeg yet again (I\u0026rsquo;m starting to wonder if OpenShot would actually bring anything to the table at this point, besides crashing)\u003c/p\u003e\n\u003ch1 id=\"plan\"\u003ePlan\u003c/h1\u003e\n\u003cp\u003eI want a video with the rear camera until the Volvo passes, then the front camera after. I want to use the front audio for the whole video.\u003c/p\u003e","title":"ZigZag Volvo"},{"content":"Just like the first two times, I\u0026rsquo;m assembling my Watkins Glen 2017 track footage with ffmpeg.\nHowever, I encountered a small issue I didn\u0026rsquo;t last year, plus I decided to change things up a bit with codecs, and audio selection.\nffmpeg requires protocol whitelist now I\u0026rsquo;m using pretty much the same concatenation command as last year (filenames are a bit different):\n$ for f in Front-*MOV; do echo file \u0026quot;$f\u0026quot;; done | ffmpeg -f concat -i - -c copy Front.MOV The error I got from ffmpeg looked like:\n[file @ 0x55d7ded86680] Protocol not on whitelist 'crypto'! [concat @ 0x55d7ded7cc80] Impossible to open 'Front-AMBA0009.MOV' file:: Invalid argument I used my google-fu and found a helpful blog that pointed me at the cause. This is actually a pretty decent security feature, and should prevent ffmpeg from reaching out to the world without your knowledge. However, it appears the default behaviour (at least from rpmfusion) differs from the man page:\nprotocol_whitelist list (input) Set a \u0026quot;,\u0026quot;-separated list of allowed protocols. \u0026quot;ALL\u0026quot; matches all protocols. Protocols prefixed by \u0026quot;-\u0026quot; are disabled. All protocols are allowed by default but protocols used by an another protocol (nested protocols) are restricted to a per protocol subset. Unless file was disallowed becayse it was used via pipe. Anyway, solution was simple enough: Whitelist \u0026lsquo;file\u0026rsquo;, and \u0026lsquo;pipe\u0026rsquo; (we\u0026rsquo;re piping in the file list).\n$ for f in Front-*MOV; do echo file \u0026quot;$f\u0026quot;; done | ffmpeg -protocol_whitelist file,pipe -f concat -i - -c copy Front.MOV Helpful options to make youtube processing faster Uploading to youtube is always a lengthy process, as youtube needs to reprocess the video. They have documented the ideal format characteristics. And I found a blog post describing the ideal youtube paramters in ffmpeg terms, so I didn\u0026rsquo;t have to read the man page again. Yay.\nThe concatenated videos (above) are just using the \u0026lsquo;copy\u0026rsquo; codecs, so I\u0026rsquo;ll leave them as-is. Only when combining the front \u0026amp; rear video will I apply the new settings:\n$ time ffmpeg -i Front.trim.MOV -filter:v \u0026quot;movie=Rear.trim.MOV, scale=480:-1 [inner]; [in][inner] overlay=1370:740 [out]\u0026quot; -codec:v libx264 -crf 21 -bf 2 -flags +cgop -pix_fmt yuv420p -codec:a aac -strict -2 -b:a 384k -r:a 48000 -movflags faststart Combined-FrontAudio.mp4 Also, you might want to do this on a powerful computer. My laptop heat throttled and was converting at 8fps. Copied the video to my server, and got 45fps. (I also had a load average of 22!!!). Your mileage will vary depending on your CPU:\nNote that hardware encoding is not an option here due to the video overlay filter.\nAudio from rear camera seems better While comparing the Front \u0026amp; Rear audio clips, it appeared that the audio from the rear camera might be a bit better. It picked up our talking a little bit less, and picked up the (rear/mid) engine noise a little bit better. Also, the front camera had an intermittant click/hiss/whine in some videos. I haven\u0026rsquo;t figured that out yet.\nI wanted to do a side-by-side comparison of front \u0026amp; rear audio, so I needed both output files. Additionally, since the bulk of the conversion time was video, and that isnt\u0026rsquo; changing, I can just swap out the audio on the video from above.\nWe\u0026rsquo;ll use -map to grab the first input video (0:v), and the second input audio (1:a):\n$ time ffmpeg -i Combined-FrontAudio.mp4 -i Rear.trim.MOV -map 0:v -map 1:a -codec:v copy -codec:a aac -strict -2 -b:a 384k -r:a 48000 -movflags faststart Combined-RearAudio.mp4 Note that I re-applied the youtube audio codec wizardry.\nWatch the video You can watch the three videos of the Watkins Glen 2017 trip on youtube\n","permalink":"/posts/ffmpeg-3/","summary":"\u003cp\u003eJust like the \u003ca href=\"https://chrisirwin.ca/posts/video-assembly-with-ffmpeg/\"\u003efirst\u003c/a\u003e \u003ca href=\"https://chrisirwin.ca/posts/ffmpeg-2/\"\u003etwo\u003c/a\u003e times, I\u0026rsquo;m assembling my Watkins Glen 2017 track footage with ffmpeg.\u003c/p\u003e\n\u003cp\u003eHowever, I encountered a small issue I didn\u0026rsquo;t last year, plus I decided to change things up a bit with codecs, and audio selection.\u003c/p\u003e\n\u003ch1 id=\"ffmpeg-requires-protocol-whitelist-now\"\u003effmpeg requires protocol whitelist now\u003c/h1\u003e\n\u003cp\u003eI\u0026rsquo;m using pretty much the same concatenation command as last year (filenames are a bit different):\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e$ for f in Front-*MOV; do echo file \u0026quot;$f\u0026quot;; done | ffmpeg -f concat -i - -c copy Front.MOV\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThe error I got from ffmpeg looked like:\u003c/p\u003e","title":"ffmpeg part three - No more Boogaloos to give"},{"content":"My Devices I decided to update the 3/4 of my Palm WebOS devices. The Pre (not pictured) and Pre2 (middle) were my primary, daily-driver phones for over two years, from September 2009 through to spring 2012, when I acquired a Galaxy Nexus and made the jump to Android.\nThe Pre3 (right) I also picked up on eBay. It came in box, with all accessories, and a spare battery.\nThe Pixi (left) I picked up cheap on eBay. \u0026ldquo;For Parts\u0026rdquo;, because it \u0026ldquo;Does not move from activation screen\u0026rdquo;. Now, the Pixi was a low-end device when it launched. It doesn\u0026rsquo;t even have WiFi. This isn\u0026rsquo;t a device you\u0026rsquo;d buy to use. But as a curiosity, I was interested. Due to the Palm servers being taken offline in January 2015, the activation process will never complete. Interestingly, this \u0026ldquo;Sprint\u0026rdquo; Pixi has a \u0026ldquo;Verizon\u0026rdquo; faceplate, possibly swapped from a Pixi Plus at some point. Model number confirms it is a Sprint device..\nThe Pre \u0026amp; Pixi are CDMA phones, locked to Bell Mobility and Sprint respectively. So they\u0026rsquo;re effectively useless.\nThe Pre2 \u0026amp; Pre3 are HSPA+ phones, and I should be able to use them on my current provider, assuming I can locate a SIM adapter.\nMetaDoctor Following the MetaDoctor Wiki is fairly straight forward.\nNote:\n\u0026ldquo;WebOS Doctor\u0026rdquo; is Palm\u0026rsquo;s official firmware update program + firmware images\n\u0026ldquo;MetaDoctor\u0026rdquo; is a community-driven makefile that alters the above \u0026ldquo;WebOS Doctor\u0026rdquo; images with user-chosen modifications.\nFetching Firmware After fetching metadoctor from github, you\u0026rsquo;re required to fetch the WebOS firmware images from Palm. Palm, of course, doesn\u0026rsquo;t exist currently. Thankfully, you can get all WebOS Doctor images via archive.org.\nYou\u0026rsquo;ll have to rename the files after download to append the WebOS version to the filename. In my case:\n# Bell Pre mv webosdoctorp100ewwbellmo.jar webosdoctorp100ewwbellmo-1.4.5.jar # Sprint Pixi mv webosdoctorp200ewwsprint.jar webosdoctorp200ewwsprint-1.4.5.jar # Unlocked Pre2 mv webosdoctorp224pre2wr.jar webosdoctorp224pre2wr-2.2.4.jar # AT\u0026amp;T Pre3 mv webosdoctorp224mantaatt.jar webosdoctorp224mantaatt-2.2.4.jar Build Modified Firmware An important resource is the MetaDoctor README file. It outlines all the options available. Myself, I just want to a few options. Instructions say to modify the Makefile, but passing args works just fine. These make commands will output a few assembled firmware images to the build directory.\n$ make DEVICE=pre CARRIER=bellmo BYPASS_ACTIVATION=1 BYPASS_FIRST_USE_APP=1 ENABLE_DEVELOPER_MODE=1 DISABLE_UPLOAD_DAEMON=1 DISABLE_UPDATE_DAEMON=1 all $ make DEVICE=pixi CARRIER=sprint BYPASS_ACTIVATION=1 BYPASS_FIRST_USE_APP=1 ENABLE_DEVELOPER_MODE=1 DISABLE_UPLOAD_DAEMON=1 DISABLE_UPDATE_DAEMON=1 all $ make DEVICE=pre2 CARRIER=wr BYPASS_ACTIVATION=1 BYPASS_FIRST_USE_APP=1 ENABLE_DEVELOPER_MODE=1 DISABLE_UPLOAD_DAEMON=1 DISABLE_UPDATE_DAEMON=1 ADD_EXTRA_CARRIERS=1 all $ make DEVICE=pre3 CARRIER=att BYPASS_ACTIVATION=1 BYPASS_FIRST_USE_APP=1 ENABLE_DEVELOPER_MODE=1 DISABLE_UPLOAD_DAEMON=1 DISABLE_UPDATE_DAEMON=1 ADD_EXTRA_CARRIERS=1 all Note that I\u0026rsquo;m using CARRIER=att for the Pre3, due to this being an AT\u0026amp;T phone. However, ADD_EXTRA_CARRIERS=1 should provide whatever additional data (APN?) for it to work on my network. The Pre2 uses CARRIER=wr. There is no \u0026ldquo;rogers\u0026rdquo; carrier, but it is on the wiki. However, it links to the webosdoctorp224pre2wr.jar file.\nInstall Firmware Interestingly, WebOS Doctor images are more than just a firmware image, like you\u0026rsquo;d have with Android (or other devices). They\u0026rsquo;re actually executable java bundles, which will push the encased firmware to the device. So you execute it on your computer. On the bright-side, they\u0026rsquo;re Java, so they should work on any platform. On the super-bright-side, they actually work just fine with the OpenJDK installed in Fedora. No need to tarnish your system with Sun/Oracle Java.\n$ java -jar build/pixi-p200eww-sprint-1.4.5/webosdoctorp200ewwsprint-1.4.5.jar Now, the down-side (of course there was one). There seems to be some trouble actually finding the USB device. It seems that it uses platform-specific builds of novacom. Novacom is sort of like both fastboot and adb utilities in the Android world. Apparently there are libusb woes here. There are instructions on chasing down old libraries to resolve this issue\u0026hellip;\nLuckily, none of that matters on Fedora 24, since novacom is actually packaged.\n$ sudo dnf install novacom Last metadata expiration check: 0:52:44 ago on Wed Sep 14 22:24:59 2016. Dependencies resolved. ================================================================================= Package Arch Version Repository Size ================================================================================= Installing: novacom x86_64 1.1.0-0.11.rc1.git.ff7641193a.fc24 fedora 10 k novacom-client x86_64 1.1.0-0.11.rc1.git.ff7641193a.fc24 fedora 24 k novacom-server x86_64 1.1.0-0.14.rc1.fc24 fedora 52 k Transaction Summary ================================================================================= Install 3 Packages Total download size: 86 k Installed size: 131 k You\u0026rsquo;ll need to run it as root to allow direct access to USB devices (You can probably configure some udev permission rules, but I won\u0026rsquo;t be doing this often enough to bother). You can instruct it to fork, but I preferred opening a second terminal, so I could \u0026lt;CTRL+C\u0026gt; the process when I\u0026rsquo;m done.\n$ sudo novacomd Update: A systemd unit is actually installed. sudo systemctl start novacomd would be the correct method to start the service\nYou should see a few lines of output indicating a device was found.\nBack in our metadoctor terminal, run the WebOS Doctor firmware we assembled earlier:\n$ java -jar build/pixi-p200eww-sprint-1.4.5/webosdoctorp200ewwsprint-1.4.5.jar A few clicks of \u0026ldquo;Next\u0026rdquo; should get your firmware flowing. Note that one of the \u0026rsquo;next\u0026rsquo; buttons will be disabled if your device isn\u0026rsquo;t found.\nDevices stuck in Activation As mentioned above, the Pixi was stuck in perpetual activation. You\u0026rsquo;ll need to force it into a mode that will accept firmware. This is sort of like bootloader mode on Android. This process should be identical for all Palm WebOS hardware.\nTurn the device off Hold Either press , or Plug in USB (both turn on the phone) The updater should see the device and enable you to load the firmware.\nAfter a successful update, the initial boot screen will be a picture of Tux with a Palm Pre.\nEnd of Pixi fun After booting up the Pixi, I noticed it immediately started in \u0026ldquo;International Roaming\u0026rdquo; mode, notified of pending voice mail, and has a phone number assigned to it.\nI\u0026rsquo;ve immediately placed it into airplane mode, then yanked the battery. Since this device lacks Wifi, there is very little reason to look at it again.\nWebOS 2 on Pre There is a process to install WebOS 2 onto WebOS 1 devices, like the Pre and Pixi. I actually used WebOS 2 on my Pre when it was my primary device back in 2011. It added some notable features, such as voice dialing (handy with bluetooth). It was perfectly functional, though a tad sluggish at times (it was targeting better hardware).\nThat said, backporting WebOS 2 to these devices is an interesting exercise. There is a WebOS 2 upgrade procedure, which involves running the appropriate script from within the metadoctor directory:\n$ scripts/meta-bellmo-pre-2.1.0 The script will list the multiple WebOS Doctor images required at the top. You\u0026rsquo;ll need to preemptively fetch these, as the URLs are bad (pointing to offline Palm servers, again). Note that they use a slightly different naming scheme (they do not get the version appended to the end).\nThe firmware install procedure is the same as above.\n","permalink":"/posts/WebOS-metadoctor/","summary":"\u003ch1 id=\"my-devices\"\u003eMy Devices\u003c/h1\u003e\n\u003cp\u003e\u003cimg alt=\"\u0026ldquo;WebOS Devices closed\u0026rdquo;\" loading=\"lazy\" src=\"/posts/WebOS-metadoctor/devices-closed.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"\u0026ldquo;WebOS Devices open\u0026rdquo;\" loading=\"lazy\" src=\"/posts/WebOS-metadoctor/devices-open.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eI decided to update the 3/4 of my Palm WebOS devices. The Pre (not pictured) and Pre2 (middle) were my primary, daily-driver phones for over two years, from September 2009 through to spring 2012, when I acquired a Galaxy Nexus and made the jump to Android.\u003c/p\u003e\n\u003cp\u003eThe Pre3 (right) I also picked up on eBay. It came in box, with all accessories, and a spare battery.\u003c/p\u003e","title":"Using Metadoctor on HP/Palm WebOS Devices"},{"content":"Note: Yes, WebOS Technically still exists. However, this article isn\u0026rsquo;t talking about the TV OS version made by LG.\nWhile this article was actually written to discuss getting Palm devices usable, I felt some preamble was necessary.\nMy Palm History I\u0026rsquo;ve got a soft spot for Palm. My first PDA was the Palm Vx, possibly the greatest PDA ever made. Easily get days of battery from a device that can store all your calendar and contacts information, synchronizing periodically with your master copy on your computer.\nPalm lost its way with later versions of PalmOS. Devices got more spec competitive, but batteries didn\u0026rsquo;t. My Tungsten T3 while technically superior in every spec, was actually less capable of performing it\u0026rsquo;s primary PIM tasks than the Vx before it.\nWe\u0026rsquo;ll just skip over the dark years:\nSoftware and Hardware divisions split Software eventually dead-ends in development hell (taking the remains of BeOS with it) Windows Mobile on Palm Hardware Eventually Palm decided to get their act together.\nHello WebOS Palm\u0026rsquo;s WebOS showing blew me away. I wanted one so much that when they finally released in Canada, I walked into a Bell store in September 2009 and signed a 3-year contract, despite being laid off work only a week earlier.\nThe biggest complaint was the Palm Prē launch hardware. While very comparable spec-for-spec with the iPhone 3G and 3GS, WebOS featured multitasking with multiple running applications. Apple didn\u0026rsquo;t. End result was the Prē was often laggy and slow. It was unfortunate.\nPalm quickly released an updated Prē Plus in May 2010 with twice the memory. Eventually, the Prē2 (released October 2010) would reveal what a WebOS device should be like. I actually had one shipped up from the United States (since Bell had started converting from CDMA to HSPA+, just in time).\nHP\u0026rsquo;s wallet to the rescue However, Palm was struggling financially. Before the Prē2 was released, HP bought Palm. This was a good thing: Palm will finally have the financial backing to take on Apple and Android platforms.\nLooking forward, the Prē3 was on the horizon for 2011. Featuring a major CPU upgrade, a significantly higher-res screen, and a refreshed design. Launching alongside the TouchPad tablet, there were some amazing demos. Tapping content between devices was mind-blowing at the time.\nAnticipation was high. Months passed.\nFinally, the Prē3 launched in Europe on August 16, 2011. \u0026ldquo;US is coming up in the near future\u0026rdquo;. Two days later, on August 18, 2011, HP kills Palm.\nThere was hope that the OS would continue, other hardware vendors might pop-up. Unfortunately, none of that happened (LG smart TVs aside).\nPalm Hardware in 2016 Due to the very open nature of WebOS (every device being a development device after a quick, official procedure) there was already a wide community modifying WebOS.\nIt was due to the efforts of these people that WebOS 2 was backported to the unsupported Prē (including features like voice dialing, etc).\nThe homebrew scene was particularly active with the \u0026ldquo;Preware\u0026rdquo; package manager, and alternative to the official App Catalog. Additionally, the nature of an OS built on HTML and Javascript opened the doors to a wide variety of patches and modifications.\nWebOS MetaDoctor allowed users to spin a new, customized WebOS image with pre-applied applications, patches, and features (including massaging hardware support). Then was then flashed to the phone for a customized user experience.\nIn late 2014, HP announced they were shutting down the authentication servers. Considering a Palm account was a required account for device activation, this had the potential to be the final nail in the WebOS coffin if it wasn\u0026rsquo;t for MetaDoctor allowing you to skip the Palm account step.\nWebOS in 2016 Luckily, HP decided to Open WebOS, as well as the Enyo app framework. This allowed others to continue development of the platform.\nLuneOS was born, and began modernizing. While this unfortunately means that it can\u0026rsquo;t run on the old Palm hardware, it does run on (slightly) newer Nexus hardware (Galaxy Nexus and Nexus 4).\nFollowups to come Stay tuned for articles on both Meta-doctoring Palm hardware, as well as running LuneOS on a Nexus 4.\n","permalink":"/posts/WebOS-in-2016/","summary":"\u003cp\u003e\u003cem\u003eNote: Yes, \u003ca href=\"http://www.techradar.com/news/television/6-best-smart-tv-platforms-in-the-world-today-1120795/4\"\u003eWebOS Technically still exists\u003c/a\u003e. However, this article isn\u0026rsquo;t talking about the TV OS version made by LG.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWhile this article was actually written to discuss getting Palm devices usable, I felt some preamble was necessary.\u003c/p\u003e\n\u003ch1 id=\"my-palm-history\"\u003eMy Palm History\u003c/h1\u003e\n\u003cp\u003eI\u0026rsquo;ve got a soft spot for Palm. My first PDA was the \u003ca href=\"https://en.wikipedia.org/wiki/Palm_Vx\"\u003ePalm Vx\u003c/a\u003e, possibly the greatest PDA ever made. Easily get days of battery from a device that can store all your calendar and contacts information, synchronizing periodically with your master copy on your computer.\u003c/p\u003e","title":"WebOS in 2016"},{"content":"Snapper is an excellent utility that provides hourly snapshots of btrfs subvolumes.\nFedora ships with selinux enabled by default. This is excellent, and shouldn\u0026rsquo;t be disabled. To allow this, most software in Fedora has appropriate rules defined, including snapper.\nHowever, snappers rules only allow it to work on / and /home. If you wish to use it to snapshot /mnt/data, or /srv, or any other particular path, you\u0026rsquo;re going to have a very bad time.\nWhile it is certainly possible to define new rules for paths you wish to back up, I decided that in this one particular case, snapper should be allowed free reign.\nsudo semanage permissive -a snapperd_t The above command tells selinux to treat snapperd_t (the context snapperd runs within) as permissive. Rule violations will still be logged, but snapper will be allowed to continue.\n","permalink":"/posts/snapperd_with_selinux/","summary":"\u003cp\u003eSnapper is an excellent utility that provides hourly snapshots of btrfs subvolumes.\u003c/p\u003e\n\u003cp\u003eFedora ships with selinux enabled by default. This is excellent, and shouldn\u0026rsquo;t be disabled. To allow this, most software in Fedora has appropriate rules defined, including snapper.\u003c/p\u003e\n\u003cp\u003eHowever, snappers rules only allow it to work on / and /home. If you wish to use it to snapshot /mnt/data, or /srv, or any other particular path, you\u0026rsquo;re going to have a \u003cstrong\u003every\u003c/strong\u003e bad time.\u003c/p\u003e","title":"Snapperd on Fedora with SELinux enabled"},{"content":"I just attended the Watkins Glen opening day for the second year. It was, again, a blast.\nI made some slight adjustments to my ffmpeg assembly procedure from last year.\nDashcam saves video in 5-minute chunks Instead of creating .list files, I simply used a pipe as input:\nfor fo in AMBA091*; do echo file \u0026quot;$fo\u0026quot;; done \\ | ffmpeg -f concat -i - -c copy Front-Track1.mov Front and Rear videos need to be combined Much like last year, I made short samples to confirm if any offsets needed to be done. However, I decided to move the video to the bottom-right corner to cover the timestamps, since they were incorrect on some videos (well, correct. Just not for this time zone)\nMath is basically the same as before for scaling, but instead of a left offset of 70, we want a right offset of 70, but using left-hand coords. Which works out to:\n1920 - 480 - 70 = 1370 After the usual synchronization samples were made, it was time to perform the final assembly.\nI used a slightly different file layout this time, keeping the front and rear videos separated. I used a loop to assemble them into a combined video:\n$ time for FILE in Track{1,2,3}; do ffmpeg -i Front/DCIM/*${FILE}*mov -vf \u0026quot;movie=Rear/DCIM/Rear-${FILE}-cat.mov, scale=480:-1 [inner]; [in][inner] overlay=1370:740 [out]\u0026quot; -strict -2 ${FILE}.mov; done You\u0026rsquo;re going to want a good CPU. This is the execution time for just under 48 minutes of video on an Intel i5-2520M:\nreal 172m2.494s user 619m51.494s sys 1m37.383s Final result You can see the resulting videos on youtube: Part 1, Part 2, and Part 3. Part 3 has some bad sound. I\u0026rsquo;m not sure why.\n","permalink":"/posts/ffmpeg-2/","summary":"\u003cp\u003eI just attended the Watkins Glen opening day for the second year. It was, again, a blast.\u003c/p\u003e\n\u003cp\u003eI made some slight adjustments to my \u003ca href=\"/posts/video-assembly-with-ffmpeg/\"\u003effmpeg assembly procedure from last year\u003c/a\u003e.\u003c/p\u003e\n\u003ch1 id=\"dashcam-saves-video-in-5-minute-chunks\"\u003eDashcam saves video in 5-minute chunks\u003c/h1\u003e\n\u003cp\u003eInstead of creating .list files, I simply used a pipe as input:\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003efor fo in AMBA091*; do echo file \u0026quot;$fo\u0026quot;; done \\\n    | ffmpeg -f concat -i - -c copy Front-Track1.mov\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1 id=\"front-and-rear-videos-need-to-be-combined\"\u003eFront and Rear videos need to be combined\u003c/h1\u003e\n\u003cp\u003eMuch like last year, I made short samples to confirm if any offsets needed to be done. However, I decided to move the video to the bottom-right corner to cover the timestamps, since they were incorrect on some videos (well, correct. Just not for this time zone)\u003c/p\u003e","title":"ffmpeg part two - Electric Boogaloo"},{"content":"I was attempting to run my laptop at a lower resolution than the laptop panel. However, by default the video is scaled to fill the panel. This causes the image to be distorted (fonts look bad, etc).\nOn Linux (with Xorg, anyway), this behaviour can be tweaked with xrandr:\n$ xrandr --output LVDS1 --set \u0026quot;scaling mode\u0026quot; \u0026quot;Center\u0026quot; This is not a persistent setting, which is fine for my purposes.\nThanks to the Arch Linux Wiki article on Intel Graphics for documenting this.\n","permalink":"/posts/scaling_mode/","summary":"\u003cp\u003eI was attempting to run my laptop at a lower resolution than the laptop panel. However, by default the video is scaled to fill the panel. This causes the image to be distorted (fonts look bad, etc).\u003c/p\u003e\n\u003cp\u003eOn Linux (with Xorg, anyway), this behaviour can be tweaked with xrandr:\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e$ xrandr --output LVDS1 --set \u0026quot;scaling mode\u0026quot; \u0026quot;Center\u0026quot;\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis is not a persistent setting, which is fine for my purposes.\u003c/p\u003e","title":"Intel GPU Scaling mode"},{"content":"In an ongoing quest to attempt to lessen my Google dependancy, I decided to self-host my Calendar and Contacts using Baïkal.\nInstalling and configuring Baïkal is sufficiently documented elsewhere. This post is a 9somewhat short) account of why I\u0026rsquo;m giving up on self-hosted contacts and calendars.\nGoogle The problems can be summed up into these bullet points:\nIt is assumed (and practically required) to use Google Play Store Google Play Store requires a Google Account Google Account means you have Mail, Calendar, and Contacts Simply adding your google account into your phone causes Mail, Calendar and Contacts to sync. Mail you can disable, and use an alternate client, as that data is housed internally to the gmail app, and not exposed system wide for other apps to use.\nCalendars and Contacts, even if you disable sync, are still \u0026ldquo;there\u0026rdquo;. Some apps might add events to the \u0026ldquo;first\u0026rdquo; calendar, without asking (which may or may not be synced to Google, and not your self-hosted calendar). Updating contacts sometimes adds those updates to the Google Contacts list. There is no apparent way to move items from one contacts/calendar account to another.\nSummary As it stands now, you can either have:\nself-hosted contacts and calendars with probably most of your data, and understand that you will miss events, and people.\nGoogle Contacts and Calendar with all of your data.\nAs much as I preferred self-hosting, it simply isn\u0026rsquo;t practically possible until you can completely remove Google Contacts/Calendars from your device, and the management apps provide the ability to move events.\n","permalink":"/posts/my_failed_experiment_with_caldav/","summary":"\u003cp\u003eIn an ongoing quest to attempt to lessen my Google dependancy, I decided to self-host my Calendar and Contacts using \u003ca href=\"http://baikal-server.com/\"\u003eBaïkal\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eInstalling and configuring Baïkal is sufficiently documented elsewhere. This post is a 9somewhat short) account of why I\u0026rsquo;m giving up on self-hosted contacts and calendars.\u003c/p\u003e\n\u003ch1 id=\"google\"\u003eGoogle\u003c/h1\u003e\n\u003cp\u003eThe problems can be summed up into these bullet points:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt is assumed (and practically required) to use Google Play Store\u003c/li\u003e\n\u003cli\u003eGoogle Play Store requires a Google Account\u003c/li\u003e\n\u003cli\u003eGoogle Account means you have Mail, Calendar, and Contacts\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSimply adding your google account into your phone causes Mail, Calendar and Contacts to sync. Mail you can disable, and use an alternate client, as that data is housed internally to the gmail app, and not exposed system wide for other apps to use.\u003c/p\u003e","title":"My failed experiment with CalDAV/CardDAV"},{"content":"I decided I wanted some stats. There are a few options: Use a service (Google Analytics, etc) or parse your logs. Both have pros and cons. This article isn\u0026rsquo;t supposed to help you decide.\nI just wanted simple stats based on logs: It\u0026rsquo;s non-intrusive to visitors, doesn\u0026rsquo;t send their browsing habits to third parties (other than what they send themselves), and uses the apache log data I\u0026rsquo;ve already got for the entire year.\nI\u0026rsquo;m mainly interested in seeing how many people actually read these articles, as well as what search terms referred them here.\nFix your logs I\u0026rsquo;ve got seven virtualhosts spread across four virtual machines. My first problem, is all were using /var/log/httpd/access_log for logging. A lot of grep work, and I managed to split those out to individual access logs: /var/log/httpd/access_log.chrisirwin.ca, for example.\nMy biggest problem is a lot of log enteries didn\u0026rsquo;t actually indicate which virtualhost they were from. I ended up spending a few hours coming up with a bunch of rules to identify all queries for my non-main virtualhosts (yay static files). Then I dumped anything that didnt\u0026rsquo; match those rules into my main virtualhost\u0026rsquo;s log (including all the generic GET / entries.\nAll my logs are sorted into per-virtualhost logs, and all lines from the original are accounted for.\nI renamed access_log to access_log.old, just so I don\u0026rsquo;t mistakenly review it\u0026rsquo;s data again.\nFix your logging Now that we\u0026rsquo;ve got separate access logs, we need to tell our virtualhosts to use them. In each virtualhost I added new CustomLog and ErrorLog definitions, using the domain name of the virtualhost.\nCustomLog \u0026quot;logs/access_log.chrisirwin.ca\u0026quot; combined ErrorLog \u0026quot;logs/error_log.chrisirwin.ca\u0026quot; Then restart httpd\n$ sudo systemctl restart httpd I also disabled logrotate, and un-rotated my logs with zcat. I\u0026rsquo;ll probably need to revisit this in the future, but 1 year worth of logs is only 55MB.\nFetch logs It goes without saying that awstats needs to be local to the logs. I have four virtual machines. Do I want to manage awstats on all of them? No.\nSo I wrote a bash script to pull in my logs to a local directory:\n$ cat /opt/logs/update-logs #!/bin/bash cd $(dirname $(readlink -f $0)) # Standard apache/httpd hosts for host in chrisirwin.ca web.chrisirwin.ca; do mkdir -p $host rsync -avz $host:/var/log/httpd/*log* $host/ done # Gitlab omnibus package is weird host=gitlab.chrisirwin.ca mkdir -p $host rsync -avz $host:/var/log/gitlab/nginx/*log* $host/ Now I have a log store with a directory per server, and logs per virtualhost within them.\nConfigure cron + ssh-keys to acquire that data, or run it manually whenever.\nInstall awstats Then I picked my internal web host, and installed awstats. This is in Fedora 22, but requires you to enable epel for CentOS/RHEL.\n$ sudo dnf install awstats And, uh, restart apache again\n$ sudo systemctl restart httpd Configure awstats Now go to /etc/awstats, and make a copy of the config for each domain:\n$ sudo cp awstats.model.conf awstats.chrisirwin.ca.conf You\u0026rsquo;ll probably want to read through all the options, but here\u0026rsquo;s all the values I modified:\nLogFile=\u0026quot;/opt/logs/chrisirwin.ca/access_log.chrisirwin.ca\u0026quot; SiteDomain=\u0026quot;chrisirwin.ca\u0026quot; HostAliases=\u0026quot;REGEX[^.*chrisirwin\\.ca$]\u0026quot; # DNSLookups is going to make log parsing take a *very* long time. DNSLookup=1 # My site is entirely https, so tell awstats that UseHTTPSLinkForUrl=\u0026quot;/\u0026quot; Run the load script Let\u0026rsquo;s just piggy-back on provided functionality:\n$ time sudo /etc/cron.hourly/awstats Mine took \u0026gt;15 minutes. I think it was primarily DNS related.\nReview your logs By default, awstats figures out what config to use based on the domain name in the URL. However, I\u0026rsquo;ve aggregated my logs to a single location. Luckily, awstats developers though of this, and you can pass along a an alternate config in the url:\nhttps://internal.chrisirwin.ca/awstats/awstats.pl?config=chrisirwin.ca Tweaks to awconfig Unless you\u0026rsquo;re running awstats on your localhost, you\u0026rsquo;ll be denied access. You\u0026rsquo;ll likely have to edit /etc/httpd/conf.d/awstats.conf and add Require ip 10.10.10.10/16, or whatever your local ip range is. Note that while you can add hostnames instead of IPs, reverse DNS needs to be configured.\nWhile there, you could also add DirectoryIndex awstats.pl.\n","permalink":"/posts/awstats_from_multiple_hosts/","summary":"\u003cp\u003eI decided I wanted some stats. There are a few options: Use a service (Google Analytics, etc) or parse your logs. Both have pros and cons. This article isn\u0026rsquo;t supposed to help you decide.\u003c/p\u003e\n\u003cp\u003eI just wanted simple stats based on logs: It\u0026rsquo;s non-intrusive to visitors, doesn\u0026rsquo;t send their browsing habits to third parties (other than what they send themselves), and uses the apache log data I\u0026rsquo;ve already got for the entire year.\u003c/p\u003e","title":"AWStats from multiple hosts"},{"content":"Update - March 09 2020 Since writing this article, I\u0026rsquo;ve made a few observations:\nMore people read this than expected QEMU has since added discard support to the standard virtio disk, avoiding the need to use virtio-scsi It kinda sorta works out of the box. I\u0026rsquo;ve written a brief followup: Discard (TRIM) with KVM Virtual Machines\u0026hellip; in 2020!\nOriginal Article I\u0026rsquo;ve got a bunch of KVM virtual machines running at home. They all use sparse qcow2 files as storage, which is nice and space efficient \u0026ndash; at least at the beginning.\nOver time, as updates are installed, temp files are written and deleted, and data moves around, the qcow2 files slowly expand. We\u0026rsquo;re not talking about a massive amount of storage, but it would be nice to re-sparsify those images.\nIn the past, I\u0026rsquo;ve made a big empty file with dd and /dev/zero, delete it, then fallocate on the host to punch the detected holes. However, this is cumbersome.\nAs it turns out, there is a better way: discard. Discard support was initially added to tell SSDs what data can be cleaned and re-used (SSDs call it \u0026lsquo;TRIM\u0026rsquo;), to preserve performance and extend drive lifetime (allowing better wear levelling). The same methods can also be used to allow a VM to tell it\u0026rsquo;s host what part of it\u0026rsquo;s storage is no longer required. This allows the host to actually regain free space when guest machines do.\nI used the following two pages as references. The first is more generically useful for machines with actual SSDs, as well as checking trim works through multiple storage layers (dm, lvm, etc).\nHow to properly activate TRIM for your SSD on Linux: fstrim, lvm and dm-crypt\nRecover Space From VM Disk Images By Using Discard/FSTRIM\nFix fstab Ensure you\u0026rsquo;re not using any device paths in fstab, like /dev/sda1 or /dev/vda1. These steps may renumber or rename your hard disks, and you don\u0026rsquo;t want to troubleshoot boot problems later on. Switch them to LABEL or UUID entries, depending on your preference/use-case.\nThis also means fixing your initrd and grub, if necessary. Most installs shouldn\u0026rsquo;t require that, though. Typically, it\u0026rsquo;s just lazy manually-added filesystems :)\nIt goes without saying, but reboot your VMs now to ensure they boot after your changes. That will make troubleshooting easier later.\nShutdown VMs and libvirtd Since I\u0026rsquo;m doing some manual munging of the VM definition files, first step is to shut down all VMs and stop libvirtd.\nUpdate machine type Some of my VMs were quite old, and were using old machine versions, as evidenced by one of the .xml files in /etc/libvirt/qemu:\n\u0026lt;type arch='x86_64' machine='pc-i440fx-1.6'\u0026gt;hvm\u0026lt;/type\u0026gt; From what I understand, machine types later than 2.1 include discard support. I wanted to update everything to the current 2.3 machine type:\nsed -e \u0026quot;s/pc-i440fx-.../pc-i440fx-2.3/\u0026quot; -i *.xml Add discard support to hard disks Your sed line will vary here. I\u0026rsquo;ve manually specified writeback caching, so my hard drive driver line looks like the following:\n\u0026lt;driver name='qemu' type='qcow2' cache='writeback'/\u0026gt; It was fairly simple to add discard:\nsed -e \u0026quot;s#writeback'/\u0026gt;#writeback' discard='unmap'/\u0026gt;#\u0026quot; -i *.xml It should now look like this:\n\u0026lt;driver name='qemu' type='qcow2' cache='writeback' discard='unmap'/\u0026gt; You could probably key off the qcow2 bit instead of the writeback bit. The order doesn\u0026rsquo;t matter.\nChange each hard drive from virtio to scsi bus All of my VMs were using virtio disks. However, they don\u0026rsquo;t pass discard through. However, the virtio-scsi controller does.\nThere is probably a pretty easy way to do this with virsh, but I opted to just use virt-manager, since I have a finite number of VMs (and reading the man page for virsh would take longer than just doing it with virt-manager).\nChange Disk Bus to SCSI:\nChange SCSI Controller to VirtIO SCSI:\nThe latter step might not be required. The only other option is \u0026ldquo;hypervisor default\u0026rdquo;, so it might just use virtio-scsi by default. Better safe than sorry.\nBoot and check your VMs After starting your VMs, you should be able to confirm that discard support is enabled:\nsudo lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE If you see 0B under DISC-MAX, then something didn\u0026rsquo;t work:\nMOUNTPOINT DISC-MAX FSTYPE / 0B ext4 However, if you see an actual size, then congrats. You support discard:\nMOUNTPOINT DISC-MAX FSTYPE / 1G ext4 Configure your VMs themselves to discard unused data Manually run an fstrim to discard all the currently unused crufty storage you\u0026rsquo;ve collected on all applicable filesystems:\nsudo fstrim -a Going forward, you can either add \u0026lsquo;discard\u0026rsquo; to the mount options in fstab, or use fstrim periodically. I opted for fstrim, as it has a systemd timer unit that can be scheduled:\nsudo systemctl enable fstrim.timer sudo systemctl start fstrim.timer Done! Or am I\u0026hellip; Now, there are additional considerations to be made during backup.\nFor example, if you use rsync, you\u0026rsquo;ll probably want to add \u0026ndash;sparse as an option, so it doesn\u0026rsquo;t inflate your backup copy to full size. However, that won\u0026rsquo;t actually punch holes that have been discarded since the last backup. So you still need to use fallocate on your backup copies to actually reclaim discarded space.\nAnother pain is I back up to a btrfs filesystem, which uses snapper to preserve previous revisions. This should be a great solution, however, there are other considerations:\nrsync\u0026rsquo;s default behaviour is to do all work in a copy, then replace the original. As far as btrfs is concerned, this is an entirely new data, and doesn\u0026rsquo;t share anything with existing snapshots. That means btrfs snapshots are quite bloated. need to use \u0026ndash;inplace to avoid above snapshot bloat. \u0026ndash;inplace and \u0026ndash;sparse are mutually-exclusive. Well shit. My current solution is to use \u0026ndash;inplace for backups, then fallocate all files. I try to manually rsync --sparse new VMs ahead of their initial backup to avoid the temporary inflation that \u0026ndash;inplace would cause.\n","permalink":"/posts/discard-with-kvm/","summary":"\u003ch1 id=\"update---march-09-2020\"\u003eUpdate - March 09 2020\u003c/h1\u003e\n\u003cp\u003eSince writing this article, I\u0026rsquo;ve made a few observations:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eMore people read this than expected\u003c/li\u003e\n\u003cli\u003eQEMU has since added discard support to the standard virtio disk, avoiding the need to use virtio-scsi\u003c/li\u003e\n\u003cli\u003eIt kinda sorta works out of the box.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI\u0026rsquo;ve written a brief followup: \u003ca href=\"/posts/discard-with-kvm-2020/\"\u003eDiscard (TRIM) with KVM Virtual Machines\u0026hellip; in 2020!\u003c/a\u003e\u003c/p\u003e\n\u003ch1 id=\"original-article\"\u003eOriginal Article\u003c/h1\u003e\n\u003cp\u003eI\u0026rsquo;ve got a bunch of KVM virtual machines running at home. They all use sparse qcow2 files as storage, which is nice and space efficient \u0026ndash; at least at the beginning.\u003c/p\u003e","title":"Discard (TRIM) with KVM Virtual Machines"},{"content":"Gnome 3 introduced a very handy feature, grouping multiple application windows (whether they be separate instances or not) into a single desktop icon. This means when \u0026lt;alt+Tab\u0026gt;ing through your windows, you can skip over the dozen firefox windows, then dive into just your terminal windows. Generally, this works great, and I think most users don\u0026rsquo;t have any issues.\nHowever, some people (myself included) use a lot of terminals. Some are temporary short-lived generic terminals. Others are long-lived running mail (mutt), or a main development session. Unfortunately, trying to switch to my email terminal can be cumbersome as I squint at thumbnails of 10+ other terminals.\nLuckily, the mechanisms to control this are somewhat accessible. Gnome matches a window to .desktop file based on some Window properties. All like matches are grouped together. Luckily, it seems GTK+ exposes functionality to modify these values at the command line. I\u0026rsquo;ve latched on to window class (WM_CLASS), but there are likely other rules that guide a match.\nWM_CLASS seems to be set by some combination of both the \u0026ndash;class and \u0026ndash;name GTK+ flags. The first step is to see if I can make an appropriate change. In gnome-terminal, I ran the following:\nyelp --class=foo --name=foo Bingo! I got a standard yelp window, but it identified itself to gnome as foo in the title bar, \u0026lt;alt+tab\u0026gt;, and dock! It didn\u0026rsquo;t get grouped with the yelp shortcut!\nLets do the same with gnome-terminal:\ngnome-terminal --class=foo --name=foo No luck. I got a new window, but it was grouped with my existing one. The above works if this is the first instance, however, so gnome-terminal is certainly capable of changing it\u0026rsquo;s window properties.\nTurns out, gnome-terminal is being clever. It\u0026rsquo;s actually running a background process, gnome-terminal-server, that owns all windows. Running an additional gnome-terminal simply pokes the existing process to create a new child window.\nIn gnome 3.6 and earlier, this was easy enough to solve. Adding --disable-factory would disable this functionality. However, since gnome 3.8, it\u0026rsquo;s now using some sort of dbus activation. I had given up on gnome-terminal.\nI tried using alternative terminals. Ultimately, I was frustrated that they didn\u0026rsquo;t pick up my theme correctly (GTK+-2.0 based), or were weird (terminator), or that I the fonts just looked better in gnome-terminal (xterm, etc), or needed a universe of dependancies (konsole)\nFinally, I came across a lead on Stack Exchange: Run true multiple process instances of gnome-terminal. This lead me to a page on Gnome\u0026rsquo;s Wiki on Running a separate instance of gnome-terminal.\nThis workaround involves manually starting a new terminal server.\n/usr/libexec/gnome-terminal-server --app-id com.example.terminal --name=foo --class=foo \u0026amp; gnome-terminal --app-id com.example.terminal Aha, so it is possible to get my desired functionality. However, while this works, it isn\u0026rsquo;t ideal. This requires running two commands, rather than just one. Additionally, the server dies after 10 seconds if no clients connect, preventing me from spawning it at login.\nBut this manual server initialization isn\u0026rsquo;t required with the standard backend. So how does that work?\nTurns out, it\u0026rsquo;s a dbus service definition. You can review the current one /usr/share/dbus-1/services/org.gnome.Terminal.service, then make our own.\nI\u0026rsquo;ve decided to call my session \u0026lsquo;PIM\u0026rsquo;, as I\u0026rsquo;m using it for my mail/calendar terminals\ncat /usr/share/dbus-1/services/org.gnome.Terminal-PIM.service [D-BUS Service] Name=org.gnome.Terminal-PIM Exec=/usr/libexec/gnome-terminal-server --class=org.gnome.Terminal-PIM --app-id org.gnome.Terminal-PIM Now, I\u0026rsquo;ve also created the associated (and like-named) .desktop file (used the /usr/share/applications/org.gnome.Terminal.desktop as a template)\n$ cat ${HOME}/.local/share/applications/org.gnome.Terminal-PIM.desktop [Desktop Entry] Name=Mail \u0026amp; PIM Comment=Mutt and Calendar Keywords=mail;mutt;calendar Exec=gnome-terminal --app-id org.gnome.Terminal-PIM -e \u0026quot;screen -DR PIM -c .screenrc-ejpim\u0026quot; Icon=mail_send Type=Application StartupNotify=true X-GNOME-SingleWindow=false Note: I\u0026rsquo;m actually launching it straight into a pre-defined screen session, but you could change the -e parameter to whatever you wish.\nBoth dbus and gnome-shell need to be restarted to pick up their changes. You can tell gnome-shell to restart itself, but the surest method I\u0026rsquo;m aware of for dbus is to log out and in again.\nNow I can run the \u0026ldquo;Mail \u0026amp; PIM\u0026rdquo; shortcut, and get a gnome-terminal window that is grouped separately (with a mail icon!).\nAt this point, it would be work investigating gnome-terminal profiles, if desired (different colour schemes, etc).\nUnfortunately, while I put my .desktop file in ~/.local/share/applications, I couldn\u0026rsquo;t find a user-specific dbus servicedir. I had to install that in the system itself with sudo. I\u0026rsquo;d prefer to have it local to my user, so I can move it easily to other systems with my existing vcsh configuration.\nUpdate 2016-10-12 Recently there was some sort of change that caused gnome-terminal-server to run via systemd service file in the users session. This means a slight change is required to the steps above.\nThe dbus service file should now read:\ncat /usr/share/dbus-1/services/org.gnome.Terminal-PIM.service [D-BUS Service] Name=org.gnome.Terminal-PIM SystemdService=gnome-terminal-server-pim.service Exec=/usr/libexec/gnome-terminal-server --class=org.gnome.Terminal-PIM --app-id org.gnome.Terminal-PIM Additionally, you\u0026rsquo;ll need to create a systemd unit file:\ncat /usr/lib/systemd/user/gnome-terminal-server-pim.service [Unit] Description=GNOME PIM Terminal Server [Service] KillMode=process Type=dbus BusName=org.gnome.Terminal-PIM ExecStart=/usr/libexec/gnome-terminal-server --class=org.gnome.Terminal-PIM --app-id org.gnome.Terminal-PIM You\u0026rsquo;ll need to systemd --user daemon-reload for the systemd changes to take effect. You may still need to log out for the dbus change.\n","permalink":"/posts/multiple-instances-of-gnome-terminal/","summary":"\u003cp\u003eGnome 3 introduced a very handy feature, grouping multiple application windows (whether they be separate instances or not) into a single desktop icon. This means when \u0026lt;alt+Tab\u0026gt;ing through your windows, you can skip over the dozen firefox windows, then dive into just your terminal windows. Generally, this works great, and I think most users don\u0026rsquo;t have any issues.\u003c/p\u003e\n\u003cp\u003eHowever, some people (myself included) use a \u003cem\u003elot\u003c/em\u003e of terminals. Some are temporary short-lived generic terminals. Others are long-lived running mail (mutt), or a main development session. Unfortunately, trying to switch to my email terminal can be cumbersome as I squint at thumbnails of 10+ other terminals.\u003c/p\u003e","title":"Multiple Instances of Gnome Terminal"},{"content":"I recently took my car to a racetrack, covered with cameras. I wanted to post these on youtube, but encountered a few issues:\nDashcam saves video in 5-minute chunks\nFront and Rear videos need to be combined\nI don\u0026rsquo;t know anything about video editing\nI didn\u0026rsquo;t have a working video editor\nFedora doesn\u0026rsquo;t seem to ship ffmpeg, and rpmfusion doesn\u0026rsquo;t support Fedora 22 yet\nThe last point was somewhat resolved by a binary build of ffmpeg.\nDashcam saves video in 5-minute chunks There is no gap or overlap with the mini0805, which makes clips easy to combine with ffmpeg\u0026rsquo;s concat filter.\nI used an input file to list all the components:\n$ cat Front-Run2.list file 'Front/DCIM/100MEDIA/AMBA1299.MOV' file 'Front/DCIM/100MEDIA/AMBA1300.MOV' file 'Front/DCIM/100MEDIA/AMBA1301.MOV' file 'Front/DCIM/100MEDIA/AMBA1302.MOV' file 'Front/DCIM/100MEDIA/AMBA1303.MOV' Then provide that to ffmpeg:\n$ ffmpeg -f concat -i Front-Run2.list -c copy Front-Run2.mov Using the \u0026lsquo;copy\u0026rsquo; codec avoids re-encoding the video, which makes this a quick operation.\nThis is likely a good time to trim the start and end of the video. I did this on the compiled version:\n$ ffmpeg -i Front-Run2.mov -ss 0:45 -t 12:30 -c copy Front-Run2.trim.mov After reviewing that output, I replace the previous version:\n$ mv Front-Run2.trim.mov Front-Run2.mov Again, the copy filter makes this a quick operation. -ss is the start offset (so the new video will start 0:45 seconds in). -t however is the duration, so it will end at the 13:15 mark of the original video. There is also a to option to set a stop time, but I missed that in the man page until writing this article.\nThis step was needed for all the videos I wanted to use.\nFront and Rear videos need to be combined This step unfortunately involved a small amount of trial-and-error.\nFirst, I wanted to overlay the rear camera footage in the bottom corner of the front camera. This required some additional information:\nI\u0026rsquo;m combining Front-Run2.mov and Rear-Run2.mov Videos are 1920x1080. I want the rear video at 25%. That\u0026rsquo;s 480x270 I want the overlay to be in from the edge. Lets say by 70 pixels. There may be a better overlay method, but this works. Now, in a perfect world, you can just compile the \u0026ldquo;finished\u0026rdquo; version. However, both of my dash cams start recording with a variance of 1-3 seconds, so the videos don\u0026rsquo;t line up exactly (even though I used the same trimming). My video has my car start moving at 1:25, so I made a 30 second video starting at 1:15.\nNote that we can\u0026rsquo;t use the \u0026lsquo;copy\u0026rsquo; codec, as we\u0026rsquo;re actually modifying the video at this point. This makes the 30 second clips a massive time saver.\n$ ffmpeg -i Front-Run2.mov -vf \u0026quot;movie=Rear-Run2.mov, scale=480:-1 [inner]; [in][inner] overlay=70:740 [out]\u0026quot; -ss 1:15 -t 30 Sample-Run2.mov I then watched the video to time the difference between when each car started moving. I found the rear video was about 1 second behind. I trimmed it using -ss 0:01, then recreated my 30 second sample. Once I was satisfied, I generated the entire video.\n$ ffmpeg -i Front-Run2.mov -vf \u0026quot;movie=Rear-Run2.mov, scale=480:-1 [inner]; [in][inner] overlay=70:740 [out]\u0026quot; Combined-Run2.mov Slight sound censoring I was discussing a few things I\u0026rsquo;d rather not have in the video. This means I needed to mute three sections of the audio.\nI watched the video, taking note of the ranges I wanted to mute:\n7:14-7:19 8:47-9:22 15:02-15:34 Unfortunately, the volume filter I was using seemed to only take seconds\u0026hellip; So:\n434-439 526-562 902-934 Now that I\u0026rsquo;ll be building my \u0026ldquo;final\u0026rdquo; video, I converted it to a \u0026ldquo;faststart\u0026rdquo; mp4 based on youtube\u0026rsquo;s video recommendations. Also, we\u0026rsquo;re not modifying the video, so we can use -vcodec copy. The audio will be re-encoded, due to the filters.\n$ ffmpeg -i Combined-Run2.mov -af \u0026quot;volume=enable='between(t,434,439)':volume=0, volume=enable='between(t,526,562)':volume=0, volume=enable='between(t,902,934)':volume=0\u0026quot; -vcodec copy -movflags faststart Final-Run2.mp4 Final result You can see the resulting video on youtube.\nOutstanding issues So all videos were 1920x1080, but youtube only offers \u0026ldquo;720p\u0026rdquo;. I don\u0026rsquo;t know why.\n","permalink":"/posts/video-assembly-with-ffmpeg/","summary":"\u003cp\u003eI recently took my car to a racetrack, covered with cameras. I wanted to post these on youtube, but encountered a few issues:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eDashcam saves video in 5-minute chunks\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eFront and Rear videos need to be combined\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eI don\u0026rsquo;t know anything about video editing\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eI didn\u0026rsquo;t have a working video editor\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eFedora doesn\u0026rsquo;t seem to ship ffmpeg, and rpmfusion doesn\u0026rsquo;t support Fedora 22 yet\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThe last point was somewhat resolved by a \u003ca href=\"http://johnvansickle.com/ffmpeg/\"\u003ebinary build of ffmpeg\u003c/a\u003e.\u003c/p\u003e","title":"Video assembly with ffmpeg"},{"content":"This is the contents of a presentation I gave to KWLug in April 2015, roughly converted to blog format. The slides are available. I\u0026rsquo;ll update this post with a link to the kwlug.org podcast when it goes online. Audio and Video available via kwlug.org.\nI didn\u0026rsquo;t get through all of the points, I barely touched on snapshots (and didn\u0026rsquo;t cover any utilities). I\u0026rsquo;ll post a follow-up with my filesystem corruption demonstration instructions.\nNote that I am not a filesystem developer, just a user who isn\u0026rsquo;t afraid to experiment and share the lessons learned. If there are any comments/corrections, email me at chris -at- chrisirwin.ca\nWith what? \u0026ldquo;Butter F S\u0026rdquo; \u0026ldquo;Better F S\u0026rdquo; \u0026ldquo;B-tree F S\u0026rdquo; \u0026ldquo;Bee Tee Arr Eff Ess\u0026rdquo; Why should I consider a new filesystem? ext3/ext4/ntfs/etc work fine for me!\nMy trip to Ottawa! BTRFS Benefits Data \u0026amp; metadata checksums Subvolumes Copy on Write Snapshots Defragmentation support Deduplication support Multi-device support (\u0026ldquo;RAID\u0026rdquo;) SSD optimizations send/receive support Data \u0026amp; metadata checksums All data is checksummed at write, and verified at read.\nThe files you save are guaranteed* to be the files you get!\nSo you don\u0026rsquo;t need ECC like ZFS? Strictly speaking, you don\u0026rsquo;t need ECC memory for either ZFS or BTRFS. ECC memory provides error detection and correction in-memory. It would be ideal to have and use, but not all machines support it.\nNot having ECC doesn\u0026rsquo;t mean you shouldn\u0026rsquo;t use a filesystem that provides checksums. It will still protect you from disk errors.\nCar analogy: If you don\u0026rsquo;t have airbags, you still wear your seat belt.\nSubvolumes One btrfs filesystem (disk/partition/etc) can contain multiple subvolumes (root, home, etc).\nLogical separation for data (like partitions or logical volumes) Can be mounted directly (like partitions or logical volumes) No division of free space (unlike partitions or logical volumes) Can be mounted as one (unlike partitions or logical volumes) While they have special abilities, subvolumes act like directories.\nCopy on Write Changes don\u0026rsquo;t physically overwrite what they\u0026rsquo;re logically overwriting.\nAAAA -\u0026gt; ABBA AAAA AAAABB [1-4] [1,5-6,4] Alternative copy on write method ex: LVM\nAAAA -\u0026gt; ABBA AAAA AAAAAA ABBAAA [1-4] [1-4] Due to having to sync the disk to save AA before writing BB, this affects write performance.\nNote: BTRFS doesn\u0026rsquo;t do this, but other things do (LVM…)\nCopy on Write tricks File copies can use no additional storage*\n$ cp --reflink=always original.jpg copy.jpg Unlike a hardlink, modifications to copy.jpg don\u0026rsquo;t touch original.jpg\nSnapshots Snapshots can be read-only or read-write.\nRead-only are ideal for facilitating backups\nRead-write can be used for testing, etc.\nSnapshots don\u0026rsquo;t use any additional disk space, and don\u0026rsquo;t snapshot free space, unlike LVM.\n$ btrfs subvolume snapshot [-r] ./home ./home-20150413 What about LVM? LVM does provide snapshots:\nNegatively affect performance and throughput \u0026ndash; one write becomes two writes and at least one sync Require a pre-allocated amount of space become corrupt when that space is used even if that space was \u0026ldquo;free\u0026rdquo; on the original volume Fragmentation! Yes, BTRFS does cause fragmentation. For this reason, BTRFS developers suggest not using databses (mysql), or virtual machines on BTRFS filesystems because they will become fragmented over time.\nThe bright side is with SSDs, fragmentation is less of an issue. SSDs themselves are already fragmented by design, so file contiguity is not important. However, a very heavily fragmented file can cause some CPU spikes when reading on an SSD.\nFor files that are not modified (images, music…), or overwritten completely (os updates, atomic file switches), there is no impact.\nDefragment support There is on-line defragmentation support, which can defragment at the filesystem, subvolume, directory, or file level.\n$ btrfs filesystem defragment [-r] [-t 5M] /home Maybe don\u0026rsquo;t defragment a SSD?. It could cause unecessary erase cycles on your SSD, but it doesn\u0026rsquo;t relocate files unneccessarily. It probably is fine with current SSDs.\nDeduplication support BTRFS supports on-line, out-of-band deduplication. This is accessed using third-party tools that look for duplicate blocks, and then tell btrfs to merge them.\nbededup dupremove I have not used these tools.\nLive deduplication support is being worked on, but will require large amounts of RAM. Large amounts of storage is typically cheaper than large amounts of RAM, but your needs may vary.\nI will not use that, either.\nDefragment and deduplication Dedup two files, then defragment them. What happens?\nDisable COW on a file You can disable COW on a directory or file. This will avoid fragmentation, but at a cost:\nData checksumming is disabled.\nSnapshots won\u0026rsquo;t work (because they rely on COW)\nYou very probably do not want to do this.\n$ chattr +C /dir/file Multi-device support (\u0026ldquo;RAID\u0026rdquo;) BTRFS implements it\u0026rsquo;s own multi-device support.\n\u0026ldquo;RAID1\u0026rdquo; just means to make sure at least two copies exist for all data. It can work with mismatched drives (1TB, 500GB, 500GB)\n\u0026ldquo;Single\u0026rdquo; means to keep only one copy of data, ensuring you can use all available space (but with no redundancy/recovery)\nNote: RAID5+6 are still in heavy development. Until recently, there was no drive replace support. I\u0026rsquo;ve not used, experimented, or investigated these modes.\nVersus traditional RAID From Data Corruption on Wikipedia:\nAs an example, ZFS creator Jeff Bonwick stated that the fast database at Greenplum – a database software company specializing in large-scale data warehousing and analytics – faces silent corruption every 15 minutes.[9] As another example, a real-life study performed by NetApp on more than 1.5 million HDDs over 41 months found more than 400,000 silent data corruptions, out of which more than 30,000 were not detected by the hardware RAID controller. Another study, performed by CERN over six months and involving about 97 petabytes of data, found that about 128 megabytes of data became permanently corrupted.\nBTRFS will detect a bad read and recover from a second copy, or cause the read to fail, hopefully making you restore your backups (rather than silently accepting data corruption)\nSimulating Data Corruption From raid.wiki.kernel.org\nRAID (be it hardware or software), assumes that if a write to a disk doesn\u0026rsquo;t return an error, then the write was successful. Therefore, if your disk corrupts data without returning an error, your data will become corrupted. This is of course very unlikely to happen, but it is possible, and it would result in a corrupt filesystem.\nSimulating Data Corruption …\nRAID cannot, and is not supposed to, guard against data corruption on the media. Therefore, it doesn\u0026rsquo;t make any sense either, to purposely corrupt data (using dd for example) on a disk to see how the RAID system will handle that. It is most likely (unless you corrupt the RAID superblock) that the RAID layer will never find out about the corruption, but your filesystem on the RAID device will be corrupted.\nThis is the way things are supposed to work. RAID is not a guarantee for data integrity, it just allows you to keep your data if a disk dies (that is, with RAID levels above or equal one, of course).\nBTRFS Integrity BTRFS supports \u0026ldquo;scrubbing\u0026rdquo;, checking the integrity of all data. This can be done on a path, or a specific device in a multi-device BTRFS.\n$ btrfs start [-B] [-c #] [-n #] \u0026lt;path\u0026gt; It can report any errors found. If another copy is available (\u0026ldquo;RAID1\u0026rdquo;), it will automatically fix the bad copy.\nDEMO! Demonstration of corruption on plain ext4, ext4 on mdadm raid 1, ext4 on mdadm raid 5, btrfs (single device), and btrfs \u0026ldquo;raid1\u0026rdquo;\nSSD optimizations BTRFS has some useful mount options\nautodefrag : Detect random writes and defragments the affected files\ndegraded : Allow you to mount a \u0026ldquo;RAID\u0026rdquo; filesystem missing a device.\nssd : \u0026ldquo;Avoiding unnecessary optimizations. This results in larger write operations and faster write throughput.\ndiscard : Enables TRIM support. This is disabled by default as it affects generational rollbacks (not covered in this talk), and many drives reserve space for garbage collection on their own. There is also some concern about \u0026ldquo;queued TRIM support\u0026rdquo;, which didn\u0026rsquo;t exist before SATA 3.1. (Also, queued TRIM support seems buggy, since Windows doesn\u0026rsquo;t use it)\nSend/Receive support BTRFS can perform it\u0026rsquo;s own backups. Much like the dump utility for ext2/3/4, this has the advantage of working at the filesystem level. It will preserve all data btrfs knows, including the checksum data, and shared blocks.\n$ btrfs send ./home-20150413 \\ | btrfs receive /mnt/backups/ My BTRFS-formatted USB backup disk now contains an exact copy of my snapshot \u0026ndash; right down to which blocks are shared between files, and the data checksums of those blocks.\nSend incremental Send/receives can also share space (and save time) as well by providing a \u0026lsquo;parent\u0026rsquo; reference that exists on both drives:\n$ btrfs send -p ./home-20150412 ./home-20150413 \\ | btrfs receive /mnt/backups/ Because all data is checksummed, you can be guaranteed to have a valid \u0026ldquo;full\u0026rdquo; home-20150413 on your backup drive.\nSend/Receive variations You can send snapshots to another host:\n$ btrfs send ./home-20150413 \\ | ssh my-server.com btrfs receive /mnt/backup/laptop/ Or if you\u0026rsquo;re backing up to ext4, S3, or another non-btrfs volume:\n$ btrfs send ./home-20150413 \\ | gzip /mnt/backup-ext4/home-20150413.gz And restore at a later date:\n$ zcat /mnt/backup-ext4/home-20150413.gz \\ | btrfs receive /path/to/restore Snapshot utilities snapper Lots of users. Weird (to me) snapshot layout\nUses some logic to keep 24 hourly snapshots, then expire those and keep daily snapshots, etc.\nNo support in itself for getting those snapshots to another disk/host\nSome issues with scheduled snapper-clean + discard\nBTRFS Wiki on Backups There are other tools, or you could roll your own cron-job.\nSo what about ZFS? I thought ZFS did all these things? Yes, it does\nWhy not just use ZFS? Go ahead\nWhy don\u0026rsquo;t you use ZFS? I don\u0026rsquo;t want to use Solaris/FreeBSD I don\u0026rsquo;t want to use a third-party filesystem on Linux I don\u0026rsquo;t want to depend on COPRs/PPAs and dkms/akmods for filesystem drivers BTRFS is built-in Why doesn\u0026rsquo;t everybody use BTRFS? It is still a young filesystem. Some features are incomplete (deduplication), dangerously incomplete (RAID5/6), or missing (live deduplication).\nIt doesn\u0026rsquo;t support all use-cases (swap files won\u0026rsquo;t work at all, fragmentation with databases/virtual machines/torrents).\nSome people have experienced problems that resulted in data loss (though not too recently). With extensive backups, this shouldn\u0026rsquo;t be an issue. However, filesystems are expected to be fool-proof.\nThere are some gotchas and quirks that are confusing.\nBTRFS \u0026ldquo;Gotchas\u0026rdquo; From Gotchas on BTRFS Wiki\ndeadlock at mount time in 3.19.1 - 3.19.3\ndeadlock on heavy (rsync) workloads in 3.15 - 3.16.1\ndefragment/snapshot problems before 3.9 - 3.14\ndefragmenting with kernels up to 2.6.37 will unlink COW-ed copies of data, don’t use it if you use snapshots, have de-duplicated your data or made copies with cp --reflink.\nBTRFS Quirks ##df is correct/incorrect/misleading\nAssuming a 1TB+500GB \u0026ldquo;RAID1\u0026rdquo;, how much free space is there?\nBTRFS Quriks Assuming a 1TB+500GB \u0026ldquo;RAID1\u0026rdquo;, how much free space is there? df reports 1.5TB.\nThere is 1.5TB of free space\nYou can only write 500GB of data to this drive. You will then get an out of space error, while df reports 500GB free space. Not intuitive or obvious.\nYou also can\u0026rsquo;t modify or overwrite files at that point, because it needs to write the new data before the old data is de-referenced and considered \u0026ldquo;free\u0026rdquo;\nBTRFS Quirks Why not have df report usable space instead of free space That\u0026rsquo;s too hard\nBTRFS Quirks What about 1x2TB + 2x500GB as \u0026ldquo;RAID1\u0026rdquo;? You could have 1.0TB of usable space: each 500GB as the \u0026ldquo;copy\u0026rdquo; for 1TB, with 1TB \u0026ldquo;free\u0026rdquo; but unusable You could have 500GB of usable space: one 500GB master, one 500GB copy, and 2TB \u0026ldquo;free\u0026rdquo; but unusable It depends on how BTRFS decides to allocate it\u0026rsquo;s data (prefer fewer drives vs spread out as many drives). It may depend on if the filesystem was grown (i.e. added 2TB drive to existing 2x500GB array). There may be other factors.\nBTRFS Quirks In theory, developers could figure out the above to list free space… But the BTRFS roadmap has \u0026ldquo;Object-level mirroring and striping\u0026rdquo;, so free space could depend on where your file was written.\nBTRFS Quirks There is no good solution to this I can think of, other than KIDS™: Keep Your Disks Simple\nShould I stay away then? I haven\u0026rsquo;t had a single BTRFS* issue that resulted in data loss* Backups solve all ext4-to-btrfs migrations are easy/hard What am I using lvm btrfs for /, /home ext4/xfs for /boot, games (steam), torrents, Virtual Machines, etc. References Official BTRFS Wiki\nBTRFS Mount options\nBTRFS Backups\nBTRFS on Arch Wiki\nSnapper article\n","permalink":"/posts/btrfs-presentation/","summary":"\u003cp\u003eThis is the contents of a presentation I gave to KWLug in April 2015, roughly converted to blog format. The \u003ca href=\"btrfs-slides.pdf\"\u003eslides are available\u003c/a\u003e. \u003cdel\u003eI\u0026rsquo;ll update this post with a link to the \u003ca href=\"http://kwlug.org\"\u003ekwlug.org\u003c/a\u003e podcast when it goes online.\u003c/del\u003e \u003ca href=\"https://kwlug.org/node/956\"\u003eAudio and Video available via kwlug.org\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eI didn\u0026rsquo;t get through all of the points, I barely touched on snapshots (and didn\u0026rsquo;t cover any utilities). I\u0026rsquo;ll post a follow-up with my filesystem corruption demonstration instructions.\u003c/p\u003e\n\u003cp\u003eNote that I am not a filesystem developer, just a user who isn\u0026rsquo;t afraid to experiment and share the lessons learned. If there are any comments/corrections, email me at chris -at- chrisirwin.ca\u003c/p\u003e","title":"Living with BTRFS"},{"content":"I recently restarted all my VMs at once and found that several managed to start services before they finished mounting their NFS shares. These shares back apache virtualhosts, mysql databases, and ultimately left several services in non-functional states. Systemd has the ability to depend on services (Requires=, and others) as well as filesystems (RequiresMountsFor=). However, I don\u0026rsquo;t want to modify or replace the .service files installed from the package. I\u0026rsquo;d have to manually reconcile changes during updates, and it just generally isn\u0026rsquo;t nice.\nTurns out systemd took this into account. From man systemd.unit:\nAlong with a unit file foo.service, a directory foo.service.d/ may exist. All files with the suffix \u0026ldquo;.conf\u0026rdquo; from this directory will be parsed after the file itself is parsed. This is useful to alter or add configuration settings to a unit, without having to modify their unit files. Make sure that the file that is included has the appropriate section headers before any directive. Note that for instanced units this logic will first look for the instance \u0026ldquo;.d/\u0026rdquo; subdirectory and read its \u0026ldquo;.conf\u0026rdquo; files, followed by the template \u0026ldquo;.d/\u0026rdquo; subdirectory and reads its \u0026ldquo;.conf\u0026rdquo; files.\nGreat, so to add directives to the httpd.service, make a .d directory to contain your additions:\n$ sudo mkdir -p /etc/systemd/system/httpd.service.d Then populate it. Since I depend on filesystems, I added a RequiresMountsFor:\n$ cat /etc/systemd/system/httpd.service.d/fix-nfs.conf [Unit] RequiresMountsFor=/var/www/sites /opt/ssl You\u0026rsquo;ll have to issue a daemon-reload to systemd for the additions to be detected.\nYou can test by umounting the directories you indicated, then systemctl start httpd.service. Systemd will note that it requires those directories, and ensure they\u0026rsquo;re mounted before starting httpd. Nice.\n","permalink":"/posts/add-dependancy-to-systemd/","summary":"\u003cp\u003eI recently restarted all my VMs at once and found that several managed to start services before they finished mounting their NFS shares. These shares back apache virtualhosts, mysql databases, and ultimately left several services in non-functional states. Systemd has the ability to depend on services (Requires=, and others) as well as filesystems (RequiresMountsFor=). However, I don\u0026rsquo;t want to modify or replace the .service files installed from the package. I\u0026rsquo;d have to manually reconcile changes during updates, and it just generally isn\u0026rsquo;t nice.\u003c/p\u003e","title":"Adding a dependency into upstream-supplied systemd units"},{"content":"So you might find yourself saying this one day:\nHELP! I just deleted a file on my btrfs! It wasn\u0026rsquo;t old enough to be in a backup, snapshot, or git It was old enough that I don\u0026rsquo;t want to retype it. How do I get it back? Also, my btrfs is on an SSD, and it might kick off a garbage collection routine at any time!\nWell, as luck would have it, I just did that to a blog post I\u0026rsquo;ve spent all evening writing.\nDon\u0026rsquo;t panic, but don\u0026rsquo;t wait. Since this is an SSD, the former location of the file was probably marked as garbage for the SSD to clean up. That said, the SSD doesn\u0026rsquo;t immediately run it\u0026rsquo;s garbage collection routine whenever you delete a random 8K file. That would be just as bad for wear as not having TRIM/discard at all.\nTypical recovery steps So, lets go through all the typical steps, and why I didn\u0026rsquo;t do them.\nUmount your drive immediately Can\u0026rsquo;t. It\u0026rsquo;s my root and home are subvolumes on that drive.\nShut down the machine, boot from live media If I was on a hard disk, I would. However, I don\u0026rsquo;t know how the firmware on my SSD works. Will it decide a powercycle is a great time to do a garbage collection sweep?\nDo some fancy btrfs generation rollback magic I don\u0026rsquo;t know how. I found Jörg Walter\u0026rsquo;s btrfs-undelete. Unfortunately, it mentions that if the drive is mounted, it might not work.\nAsk on IRC or mail list or reddit Remember, this file was in my $HOME, is currently \u0026ldquo;unallocated\u0026rdquo; space that could be overwritten or garbage collected at any time. \u0026ldquo;Sooner\u0026rdquo; is really better than \u0026ldquo;later\u0026rdquo;.\nHow I got my file back So I remembered I had referenced the path \u0026ldquo;/var/www/ssl\u0026rdquo; in the file. This is important, because:\nIt\u0026rsquo;s short, and I know it\u0026rsquo;s in the file. Trying to remember a longer string potentially means not finding your file because your grammar or punctuation is incorrect. This path does not exist on this computer or any other files on this computer, so I should only find copies of this file. While I\u0026rsquo;m not umounting the device, it would be stupid to deliberately write data to it. I already had a second drive connected, so I used it for recovery:\n$ cd /my/other/drive Now, I\u0026rsquo;m going to grep the raw block device for my short, unique string. I\u0026rsquo;m going to tell grep to output 1000 characters of context (thats 1000 characters before and after a match), and write that to a file. This needs to be done as root, obviously.\n$ grep --text -C 1000 \u0026quot;/var/www/ssl\u0026quot; /dev/mapper/vg_w520-btrfs_ssd \u0026gt; pleasework.txt Once it finishes:\n$ ls -lh pleasework.txt -rw-r--r--. 1 root root 92M 2015-01-16 03:02 pleasework.txt Oh boy. 92MB of matches for a 8K file. Lots of matches, since I :w after every sentence. vim isn\u0026rsquo;t going to like such a big file\u0026hellip;\nThat said, I couldn\u0026rsquo;t think of a tool I\u0026rsquo;d rather use than vim for searching through the file. When all you have is a hammer, every problem looks like a nail, after all. After opening it in vim, I I searched for instances of \u0026ldquo;/var/www/ssl\u0026rdquo;, and found that there were many, many instances. However, many had been partially overwritten, as in this screenshot below:\nBut I know my copy is in there, so now I\u0026rsquo;ll search for a more recent string. One that I typed in one of my most recent edits, just before my mv fiasco. Searching for this string found three copies:\npretty similar to my article covering [[SELINUX and apache (httpd) They were all very close, but in my case, only one wasn\u0026rsquo;t corrupted, which is the copy I extracted. Otherwise, I\u0026rsquo;d have saved all copies and compared them.\nLessons Learned Backups, snapshots, and git won\u0026rsquo;t help you if files are too young to be in any of them. You can potentially recover a deleted file, quickly, using just tools you have on-hand (grep \u0026amp; vim). Be more careful when doing file operations. When doing a mv *.mdwn, be sure you type the destination. And if you think you typed ls, double check. Or make sure the file you care about is alphabetically first, at least ;) Caveats Lots of things could have prevented this from working. I think I was only able to recover due to:\nfilesystem behaviour vim\u0026rsquo;s atomic file saves (so writing out a whole fresh copy of the file every time) SSD being mostly empty (potentially avoiding aggressive garbage collection cycles) Or put more simply: I got lucky.\nYour mileage may vary.\nNo warranties.\nBest of luck.\n","permalink":"/posts/recovering-a-deleted-file-on-btrfs/","summary":"\u003cp\u003eSo you might find yourself saying this one day:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eHELP! I just deleted a file on my btrfs!\nIt wasn\u0026rsquo;t old enough to be in a backup, snapshot, or git\nIt was old enough that I don\u0026rsquo;t want to retype it.\nHow do I get it back?\nAlso, my btrfs is on an SSD, and it might kick off a garbage collection routine at any time!\u003c/p\u003e\u003c/blockquote\u003e\n\u003cp\u003eWell, as luck would have it, I just did that to \u003ca href=\"/posts/fedora-cloud-image-on-linode\"\u003ea blog post I\u0026rsquo;ve spent all evening writing\u003c/a\u003e.\u003c/p\u003e","title":"Recovering a deleted file (on btrfs)"},{"content":"Linode offers pre-assembled Fedora VMs, but their environment doesn\u0026rsquo;t support SELINUX. You also don\u0026rsquo;t get any notification on when you should reboot for new kernels, etc.\nI decided to attempt to adapt Fedora\u0026rsquo;s stock cloud image on Linode. It was not without it\u0026rsquo;s own effort, but ultimately I think it\u0026rsquo;s a better solution than attempting to retro-fit the linode image for booting. Also, I\u0026rsquo;m using the fedora cloud image on all my other VMs, so I\u0026rsquo;m familiar with how it\u0026rsquo;s set up (and can easily spool up a local copy for testing).\nThis guide also covers setting up a small raid1 btrfs filesystem for my data storage, as well as setting selinux context for httpd.\nPreparation There are some quirks for booting your own kernels on Linode. It does not support your own bootloader \u0026ndash; you have to use their supplied version of grub-legacy (called pv_grub). Unfortunately, it seems this version of grub doesn\u0026rsquo;t seem to like partitions, so we\u0026rsquo;ve got create our own /boot.\nLuckily, Fedora\u0026rsquo;s Cloud image ships a grub-legacy config that will (almost) work out of the box. After a slight modification, it will happily update correctly when kernel-updates are installed, requiring no future meddling.\nLinode also doesn\u0026rsquo;t have a way to upload disk images, so we need a temporary VM to bootstrap our new system.\nWe will require a pre-existing Linode VM. Either deploy one, or use your current one. Create one 4GB raw \u0026ldquo;root\u0026rdquo; disk for your new fedora system Create one 500mb ext4 \u0026ldquo;boot\u0026rdquo; disk for /boot. This is important, you need a linode-formatted disk for /boot (possibly a raw disk without a partition table will work too) Create one 100mb raw \u0026ldquo;cloud-init\u0026rdquo; disk for bootstrapping Create two 5GB raw disks for btrfs storage, to be used later. Attach the disks to your pre-existing (or temporary) VM. I used the following disk IDs:\nxvdd -\u0026gt; root xvde -\u0026gt; boot xvdf -\u0026gt; cloud-init Installation Boot your VM.\nWe\u0026rsquo;re going to need some additional tools that were not available on my pre-existing Fedora 20 system. Install them:\n$ yum install dosfstools cloud-utils cloud-utils-growpart genisoimage After downloading the Fedora cloud image, use dd to put the image on our root disk. Remember, the image is compressed when downloaded.\n$ xzcat Fedora-Cloud-Base-20141203-21.x86_64.raw.xz | dd of=/dev/xvdd Expand the root partition and filesystem to fill the 4GB disk:\n$ growpart /dev/xvdd 1 $ e2fsck -f /dev/xvdd1 $ resize2fs /dev/xvdd1 mount it and set up our separate /boot\n## Mount new image $ mkdir /mnt/f21 $ mount /dev/xvdd1 /mnt/f21 ## Move boot directory out of the way $ mv /mnt/f21/boot /mnt/f21/boot-dist ## Make new boot directory $ mkdir /mnt/f21/boot $ chmod 000 /mnt/f21/boot ## Mount new boot filesystem (remember: no partitions, and pre-formatted by linode) $ mount /dev/xvde /mnt/f21/boot ## Copy boot contents $ cp -a /mnt/f21/boot-dist/* /mnt/f21/boot Another limitation of pv_grub is that it\u0026rsquo;s hard-coded to look for the grub configuration in /boot on our bare disk. Since the disk only contains /boot, it won\u0026rsquo;t find it. We can fix this using a recursive symlink.\n$ ln -sfn . /mnt/f21/boot/boot Since we\u0026rsquo;re don\u0026rsquo;t have a partition on our boot device, edit /mnt/f21/boot/grub/menu.lst to use (hd0) instead of (hd0,0)\nUpdate /mnt/f21/etc/fstab. I prefer to use UUIDs for system disks. If I ever need to mount another system\u0026rsquo;s disk (F22 in the future?), I\u0026rsquo;d hate to have multiple filesystems labelled \u0026ldquo;boot\u0026rdquo; or \u0026ldquo;fedora\u0026rdquo;. Additionally, this way I don\u0026rsquo;t care what xvdX disk each filesystem is connected to.\n# Find UUID for our /boot $ blkid /dev/xvde /dev/xvde: UUID=\u0026quot;65235d57-6e89-4347-b23e-77d0a365284c\u0026quot; TYPE=\u0026quot;ext4\u0026quot; # Add it to fstab. You can use vi, or echo it. Be sure to \u0026gt;\u0026gt; append, not \u0026gt; replace! $ echo \u0026quot;UUID=65235d57-6e89-4347-b23e-77d0a365284c /boot ext4 defaults 1 1\u0026quot; \u0026gt;\u0026gt; /mnt/f21/etc/fstab Make cloud-init disk. I installed dosfstools because cloud-init supposed to support simple vfat disks, and Linode can\u0026rsquo;t add a cdrom device. vfat doesn\u0026rsquo;t work, but iso9660 on a block device does. Use cloud-localds create an iso9660-formatted disk. Have your user-data and meta-data files ready.\ncloud-localds /dev/xvdf ~/user-data ~/meta-data We now have all the necessary bits to perform our first-boot on the new Fedora 21 system.\nFirst Boot Make new VM config for your Fedora 21 system.\nxvda -\u0026gt; boot xvdb -\u0026gt; root xvdc -\u0026gt; cloud-init Boot VM, let it do it\u0026rsquo;s thing. I was monitoring the console from Linode\u0026rsquo;s LISH, so I could see when cloud-init finished, then I removed cloud init and upgraded the system.\n$ yum remove cloud-init $ yum upgrade After shutting down my VM, I removed the cloud-init disk from the VM.\nInstall Software What good is a server that doesn\u0026rsquo;t do anything? Install the stuff I need.\n$ yum install httpd screen vim btrfs-progs BTRFS While we\u0026rsquo;re editing the VM definition, add our two 5gb raw disks for btrfs. I\u0026rsquo;m using two disks so btrfs can correct checksum errors. It can detect errors with only one disk, but needs a second copy to actually fix the errornous data.\nxvdd -\u0026gt; btrfs-disk-1 xvde -\u0026gt; btrfs-disk-2 Boot the VM, then make our filesystem with raid1 redundancy. Then mount it.\n$ mkfs.btrfs -L linode_storage -d raid1 /dev/xvdd /dev/xvde $ mkdir -p /mnt/btrfs/linode_storage $ mount LABEL=linode_storage /mnt/btrfs/linode_storage I\u0026rsquo;m going to need some subvolumes for my various storage needs:\n$ btrfs subvol create /mnt/btrfs/linode_storage/home $ btrfs subvol create /mnt/btrfs/linode_storage/sites $ btrfs subvol create /mnt/btrfs/linode_storage/ssl Add fstab entries. I like using LABELs for data disks since they\u0026rsquo;re not machine-specific. I\u0026rsquo;ll only have one linode_storage, so there\u0026rsquo;s no conflict issue. We need to add the degraded bit in case one of our disks goes completely south. The last thing you want is systemd sitting at the boot console waiting for you to press CTRL-D because your storage is incomplete (but recoverable).\nLABEL=linode_storage /mnt/btrfs/linode_storage btrfs degraded 0 0 LABEL=linode_storage /var/www/sites btrfs degraded,subvol=sites 0 0 LABEL=linode_storage /home btrfs degraded,subvol=home 0 0 LABEL=linode_storage /var/www/ssl btrfs degraded,subvol=ssl 0 0 Mount the subvolumes. You may need to play some musical chairs with /home, and any other directories you\u0026rsquo;re replacing.\n$ mv /home /home-dist $ mkdir /home $ chmod 0000 /home $ mkdir /var/www/ssl $ chmod 0000 /var/www/ssl $ mkdir /var/www/sites $ chmod 0000 /var/www/sites $ mount -a $ mv /home-dist/* /home/ Configure httpd This is pretty similar to my article covering SELINUX and apache (httpd), so I\u0026rsquo;m not going to get into the details of configuring httpd. So ensure your ownership, permissions, and config are good at this point.\nPermissions aside, we also need to set our SELINUX contexts for the new directories. I\u0026rsquo;m not using NFS on this system, so I\u0026rsquo;m actually properly (well, hopefully) configuring selinux instead of setting a blanket context on the mount point.\nUse semanage to add a file context type to our new site and ssl directories. Then use restorecon to apply it. You could use chcon directly, but if a system-wide relabel happens, you\u0026rsquo;ll lose your changes.\n# http sites $ semanage fcontext -a -t httpd_sys_content_t \u0026quot;/var/www/sites(/.*)?\u0026quot; $ restorecon -R /var/www/sites # SSL certs $ semanage fcontext -a -t cert_t \u0026quot;/var/www/ssl(/.*)?\u0026quot; $ restorecon -R /var/www/ssl I recommend using firewalld and fail2ban, but I\u0026rsquo;m not going to cover that here. That said, I will say not to forget to poke a hole for your web server:\n$ firewall-cmd --permanent --add-service=http $ firewall-cmd --permanent --add-service=https $ firewall-cmd --reload Start apache, and you should be good.\n$ systemctl start httpd Remember, ausearch -m avc is your best friend to diagnose access errors caused by selinux\n","permalink":"/posts/fedora-cloud-image-on-linode/","summary":"\u003cp\u003eLinode offers pre-assembled Fedora VMs, but their environment doesn\u0026rsquo;t support SELINUX. You also don\u0026rsquo;t get any notification on when you should reboot for new kernels, etc.\u003c/p\u003e\n\u003cp\u003eI decided to attempt to adapt Fedora\u0026rsquo;s stock cloud image on Linode. It was not without it\u0026rsquo;s own effort, but ultimately I think it\u0026rsquo;s a better solution than attempting to retro-fit the linode image for booting. Also, I\u0026rsquo;m using the fedora cloud image on all my other VMs, so I\u0026rsquo;m familiar with how it\u0026rsquo;s set up (and can easily spool up a local copy for testing).\u003c/p\u003e","title":"Fedora Cloud Image on Linode"},{"content":"I\u0026rsquo;ve just built a new web server vm, basically identical to my mariadb one, and the fedora cloud image. This is documentation on how I configured it, as well as the ttrss update daemon.\nTo get nfs to work, install nfs-utils. I need some packages also for ttrss.\n$ yum install nfs-utils httpd php php-mysql php-mbstring php-xml I\u0026rsquo;m putting the web root on an nfs mount from my nas. I have multiple virtualhosts.\n/etc/httpd/conf.d/ttrss.chrisirwin.ca:\n# Bind only this IP address \u0026lt;VirtualHost *:80\u0026gt; ServerAdmin webmaster@chrisirwin.ca ServerName ttrss.chrisirwin.ca DocumentRoot /var/www/sites/ttrss.chrisirwin.ca \u0026lt;Directory /\u0026gt; Options +FollowSymLinks +MultiViews -Indexes AllowOverride None \u0026lt;/Directory\u0026gt; \u0026lt;Directory /var/www/sites/ttrss.chrisirwin.ca/\u0026gt; Options +FollowSymLinks +MultiViews +ExecCGI -Indexes AllowOverride All Order allow,deny allow from all Require all granted \u0026lt;/Directory\u0026gt; \u0026lt;/VirtualHost\u0026gt; Make the sites directory, and restrict it\u0026rsquo;s permissions:\n$ mkdir -p /var/www/sites $ chown root:root /var/www/sites $ chmod 0000 /var/www/sites Add an fstab entry. For selinux, we need the httpd_sys_script_exec_t context to allow server-side code to run:\nnas:/hosts/web/var/www/sites /var/www/sites nfs4 nosharecache,context=\u0026quot;system_u:object_r:httpd_sys_script_exec_t:s0\u0026quot; 0 0 Finally, we need to tell selinux that we\u0026rsquo;re okay with our scripts connecting to a network sql server:\n$ setsebool -P httpd_can_network_connect_db on Now, unfortunately, ttrss won\u0026rsquo;t work. It needs r/w in some of it\u0026rsquo;s directories, and selinux won\u0026rsquo;t allow write in the script context. However, we can move those write directories elsewhere, using a different context. Luckily, the context of a symlink is determined by it\u0026rsquo;s destination!\n$ mkdir -p /var/www/data $ chown root:root /var/www/data $ chmod 0000 /var/www/data Add another fstab entry, but with httpd_sys_rw_content_t context\nnas:/hosts/web/var/www/data /var/www/data nfs4 nosharecache,context=\u0026quot;system_u:object_r:httpd_sys_rw_content_t:s0\u0026quot; 0 0 We need to move and symlink the rw directories:\nfor item in cache feed-icons lock; do $ mv /var/www/sites/ttrss.chrisirwin.ca/${item} /var/www/data/ttrss.chrisirwin.ca/${item} $ ln -sfn /var/www/sites/ttrss.chrisirwin.ca/${item} /var/www/data/ttrss.chrisirwin.ca/${item} fi Now, enable httpd, and start the service:\n$ systemctl enable httpd $ systemctl start httpd If you\u0026rsquo;re having issues starting the service, you can check for any recent selinux denials.\n$ ausearch -m avc | tail Finally, the update daemon. I\u0026rsquo;m using the following service file, /etc/systemd/ttrss.chrisirwin.ca.service:\n[Unit] Description=Tiny Tiny RSS update daemon # Can't depend on mysqld/mariadb, since it's on another host. #After=network.target mysqld.service After=network.target #Requires=mysqld.service [Service] User=apache Group=apache WorkingDirectory=/var/www/sites/ttrss.chrisirwin.ca Type=simple StandardOutput=null StandardError=syslog ExecStart=/bin/php ./update_daemon2.php PrivateTmp=true InaccessibleDirectories=/home /root /boot /mnt /media ReadOnlyDirectories=/etc /usr [Install] WantedBy=multi-user.target Now, enable ttrss, and start the service:\n$ systemctl enable /etc/systemd/ttrss.chrisirwin.ca.service $ systemctl start ttrss If you edit the service file after installing it, you\u0026rsquo;ll need to reload systemd for it to re-parse the service file:\n$ systemctl daemon-reload ","permalink":"/posts/selinux-and-apache-httpd/","summary":"\u003cp\u003eI\u0026rsquo;ve just built a new web server vm, basically identical to my \u003ca href=\"/posts/selinux-and-mariadb-mysql\"\u003emariadb one\u003c/a\u003e, and the \u003ca href=\"/posts/fedora-cloud-for-mere-mortals\"\u003efedora cloud image\u003c/a\u003e. This is documentation on how I configured it, as well as the ttrss update daemon.\u003c/p\u003e\n\u003cp\u003eTo get nfs to work, install \u003ccode\u003enfs-utils\u003c/code\u003e. I need some packages also for ttrss.\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e$ yum install nfs-utils httpd php php-mysql php-mbstring php-xml\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI\u0026rsquo;m putting the web root on an nfs mount from my nas. I have multiple virtualhosts.\u003c/p\u003e","title":"SELinux and apache (httpd)"},{"content":"I\u0026rsquo;ve just built a new mysql server vm, using the instructions I posted previously. This is documentation on how I configured it.\nTo get nfs to work, install nfs-utils.\n$ yum install nfs-utils httpd mariadb mariadb-server I\u0026rsquo;m putting the database on an nfs mount from my nas. Socket files can\u0026rsquo;t exist on nfs. It\u0026rsquo;s easier to move the data than the socket file (I think I hit an selinux issue with socket access).\nAdd the following to the /etc/my.cnf:\n[mysqld] # Move datadir to mysql-data datadir=/var/lib/mysql-data # Default socket socket=/var/lib/mysql/mysql.sock # Wasn't binding to an external ipv4 by default bind-address=10.10.11.107 Make the mysql-data directory, and restrict it\u0026rsquo;s permissions:\n$ mkdir -p /var/lib/mysql-data $ chown root:root /var/lib/mysql-data $ chmod 0000 /var/lib/mysql-data Now, mysql-data files will need to have the mysqld_db_t selinux context applied to them. Those don\u0026rsquo;t carry over nfs, so we can force a context in /etc/fstab:\nnas:/hosts/sql/var/lib/mysql /var/lib/mysql-data nfs4 nosharecache,context=\u0026quot;system_u:object_r:mysqld_db_t:s0\u0026quot; 0 0 Now, enable mariadb, and start the service:\n$ systemctl enable mariadb $ systemctl start mariadb If you\u0026rsquo;re having issues starting the service, you can check for any recent selinux denials.\n$ ausearch -m avc | tail ","permalink":"/posts/selinux-and-mariadb-mysql/","summary":"\u003cp\u003eI\u0026rsquo;ve just built a new mysql server vm, using the instructions I \u003ca href=\"/posts/fedora-cloud-for-mere-mortals\"\u003eposted previously\u003c/a\u003e. This is documentation on how I configured it.\u003c/p\u003e\n\u003cp\u003eTo get nfs to work, install \u003ccode\u003enfs-utils\u003c/code\u003e.\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e$ yum install nfs-utils httpd mariadb mariadb-server\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eI\u0026rsquo;m putting the database on an nfs mount from my nas. Socket files can\u0026rsquo;t exist on nfs. It\u0026rsquo;s easier to move the data than the socket file (I think I hit an selinux issue with socket access).\u003c/p\u003e","title":"SELinux and mariadb (mysql)"},{"content":"I\u0026rsquo;ve come across a post by Marek Goldmann detailing the basics of running Fedora cloud images on kvm. I found it was a massive help, but only got me part-way toward what I wanted.\nThe documentation for cloud-init is sub-par from what I can find, so I thought I\u0026rsquo;d post this with my own modifications.\nMy requirements were:\nEasy scriptable deployment (Marek\u0026rsquo;s script got me 100% of this goal) Static (sequential) IP assignments instead of DHCP hostname != KVM domain name, making system replacements easier Additionally, I changed some logic\nChanged user-data to an external file, rather than a big echo line Generic filenames (my-vm/root.qcow2 instead of my-vm/my-vm.qcow2) Don\u0026rsquo;t destroy \u0026amp; remove VMs with a conflicting name Leave the bootstrapping files behind, in case I need to re-create the machine later (or I alter the template and want to revisit) Changed some logging Files (and hierarchy ) look like this:\nvirt-install-fedora.sh skel/next-ip skel/user-data When these are booted with my user-data, there is no password (R means random, but doesn\u0026rsquo;t seem to work), which effectively means you can\u0026rsquo;t log in to it with a password. There is an authorized key for the default user (cirwin), which means I can still ssh to these machines.\nThe virt-install-fedora.sh script incriments next-ip, and then uses sed to replace XXIP with the incrimented ip. So technically, this will only work up to 255 times before it breaks (probably 253 for a /24 subnet, and you may have conflicts).\nSome unresolved issues:\nNetwork config is obviously suboptimal. There is a network module for cloud-init, but I couldn\u0026rsquo;t get it to work (it may not have been ported to Fedora) I think systemd kills cloud-init if it runs for more than a minute or so, making yum install ... risky, and yum upgrade is right out Ultimately, I\u0026rsquo;ve started using these for my in-house experimental machines. A new one can be ready to in about two minutes.\n","permalink":"/posts/fedora-cloud-for-mere-mortals/","summary":"\u003cp\u003eI\u0026rsquo;ve come across a post by Marek Goldmann detailing the basics of \u003ca href=\"https://goldmann.pl/blog/2014/01/16/running-fedora-cloud-images-on-kvm/\"\u003erunning Fedora cloud images on kvm\u003c/a\u003e. I found it was a massive help, but only got me part-way toward what I wanted.\u003c/p\u003e\n\u003cp\u003eThe documentation for cloud-init is sub-par from what I can find, so I thought I\u0026rsquo;d post this with my own modifications.\u003c/p\u003e\n\u003cp\u003eMy requirements were:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eEasy scriptable deployment (Marek\u0026rsquo;s script got me 100% of this goal)\u003c/li\u003e\n\u003cli\u003eStatic (sequential) IP assignments instead of DHCP\u003c/li\u003e\n\u003cli\u003ehostname != KVM domain name, making system replacements easier\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAdditionally, I changed some logic\u003c/p\u003e","title":"Fedora cloud for mere mortals"},{"content":"or \u0026ldquo;That\u0026rsquo;s why you read changelogs\u0026rdquo;\nI was having some significant difficulties with offlineimap after upgrading to 6.5.4 (I\u0026rsquo;m using Ubuntu currently, so I upgraded straight from 6.3.4).\nThe original issue was a FolderValidity error, which is covered by the offlineimap FAQ. This affected my \u0026ldquo;All Mail\u0026rdquo; folder (It also affected my \u0026ldquo;Sent Mail\u0026rdquo;, but it got lost in the noise of the 67964 messages in All Mail). The recovery process is simply to remove that folder (and it\u0026rsquo;s sync history), and start over. Unfortunate, but fine. That\u0026rsquo;s why I\u0026rsquo;ve got a 28Mbit connection, after all.\nOfflineimap happily re-downloads my messages:\n$ du -sh \u0026quot;All Mail\u0026quot; 1.2G All Mail $ find \u0026quot;All Mail\u0026quot; -type f | wc -l 67964 However, the second sync deletes all my messages! (Only locally in this case! Remember to keep ~/.maildir in git!).\nThe third sync downloads them all again. WTF.\nSeveral hours and gigabytes of troubleshooting later, it turns out that OfflineIMAP 6.4 (which I had skipped) introduced the ability to create remote folders. Previously, only folders that existed on the remote would be synced. Local-only folders were effectively an island. Nothing here should cause an issue.\nHowever, I was using a feature called \u0026ldquo;nametrans\u0026rdquo;. Nametrans is applied at the \u0026ldquo;Repository\u0026rdquo; level (Remember: Offlineimap accounts consist of two repositories: typically a local and remote). My nametrans rule was configured in the remote gmail repository.\nnametrans = lambda foldername: re.sub('^(\\[Gmail\\]/)', '', foldername) The result of this being that the \u0026ldquo;[Gmail]\u0026rdquo; prefix applied to Gmail\u0026rsquo;s built-in labels is removed when syncing the folder\u0026rsquo;s name. For example, the remote folder \u0026ldquo;[Gmail]/All Mail\u0026rdquo; gets translated to \u0026ldquo;All Mail\u0026rdquo; before being synced with your local repository.\nRemote:\u0026quot;[Gmail]/All Mail -\u0026gt; nametrans -\u0026gt; Local:\u0026ldquo;All Mail\u0026rdquo; The change is that now non-existent remote folders can now be created. In previous versions, this would never have occured:\nLocal:\u0026ldquo;All Mail -\u0026gt; nametrans -\u0026gt; Remote:\u0026ldquo;All Mail\u0026rdquo; As you can see, a new remote folder called \u0026ldquo;All Mail\u0026rdquo; gets created.\nThe problem here becomes obvious: One local folder effectively maps to two remotes, and both are conflicting (one is empty, one is full)\nIf I had actually read the changelog, I would have seen this under \u0026ldquo;OfflineIMAP v6.3.5-rc2\u0026rdquo;:\nFolders will now also be automatically created on the REMOTE side of an account if they exist on the local side. Use the folderfilters setting on the local side to prevent some folders from migrating to the remote side. Also, if you have a nametrans setting on the remote repository, you might need a nametrans setting on the local repository that leads to the original name (reverse nametrans).\nThe giant red warning on the nametrans documentation also helps, if you know what you\u0026rsquo;re looking for.\nThe \u0026ldquo;reverse nametrans\u0026rdquo; for my local folder is:\nnametrans = lambda foldername: re.sub('^(Sent Mail|Drafts|Trash|All Mail|Spam)$', '[Gmail]/\\\\1', foldername) Also note that my configuration is assuming sep = /. Also note that if your language is not en_us (in the website), your folder names will be in your local language or terms (\u0026ldquo;Bin\u0026rdquo; instead of \u0026ldquo;Trash\u0026rdquo; for en_uk, etc.).\nYou\u0026rsquo;ll want to remove the duplicate labels on the Gmail side before attempting to sync, otherwise you\u0026rsquo;ve not fixed the issue (multiple remotes map to one local). Gmail actually prefixes these bad folders with \u0026ldquo;[imap]\u0026rdquo;, which is nice.\nIdeally, offlineimap would do two additonal things:\nWarn if a nametrans is only applied to one side.\nPre-calculate and preserve the nametrans mappings before performing the sync, so a single folder can\u0026rsquo;t be referenced twice.\n","permalink":"/posts/offlineimap-deletes-everything/","summary":"\u003cp\u003eor \u0026ldquo;That\u0026rsquo;s why you read changelogs\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eI was having some significant difficulties with offlineimap after upgrading to 6.5.4 (I\u0026rsquo;m using Ubuntu currently, so I upgraded straight from 6.3.4).\u003c/p\u003e\n\u003cp\u003eThe original issue was a FolderValidity error, which is covered by the \u003ca href=\"http://docs.offlineimap.org/en/latest/FAQ.html#what-is-the-uid-validity-problem-for-folder\"\u003eofflineimap FAQ\u003c/a\u003e. This affected my \u0026ldquo;All Mail\u0026rdquo; folder (It also affected my \u0026ldquo;Sent Mail\u0026rdquo;, but it got lost in the noise of the 67964 messages in All Mail). The recovery process is simply to remove that folder (and it\u0026rsquo;s sync history), and start over. Unfortunate, but fine. That\u0026rsquo;s why I\u0026rsquo;ve got a 28Mbit connection, after all.\u003c/p\u003e","title":"Offlineimap repeatedly deleting All Mail"},{"content":"I recently got a new machine for work, with lots of fancy memory and everything. First thing to do was install a friendly operating system with gnome-shell, and set up a VM for my windows/office requirements at work.\nNormally I would favour Virt-Manager+kvm, but since we\u0026rsquo;re now using vmware-server (and soon vsphere hypervisor) I thought it would be better to keep compatible. Luckilly, vmware has a free player that works pretty well for my basic needs.\nUnfortunately, there seems to be a conflict between vmware-player and gnome-shell, that occasionally causes the enture UI to completely hang. Switching to a console and issuing pkill -9 gnome-shell was sufficient to get the system to kick itself back to functioning again. However, this was extremely annoying.\nI finally found a workaround: Setting a hidden preference in vmware to allow a VM to continue running after the player program has quit, then using RDP to connect to my windows VM (I could probably also set up VNC console access via the vm config as well)\nThe preferences file is \u0026ldquo;~/.vmware/preferences\u0026rdquo;. Add the following line:\npref.vmplayer.exit.vmAction = \u0026quot;disconnect\u0026quot; Configuring RDP to your windows VM (or VNC through vmware) is documented elsewhere, and left as an exercise to the reader.\n","permalink":"/posts/workaround-for-conflict-between-gnome-shell-a/","summary":"\u003cp\u003eI recently got a new machine for work, with lots of fancy memory and everything. First thing to do was install a friendly operating system with gnome-shell, and set up a VM for my windows/office requirements at work.\u003c/p\u003e\n\u003cp\u003eNormally I would favour Virt-Manager+kvm, but since we\u0026rsquo;re now using vmware-server (and soon vsphere hypervisor) I thought it would be better to keep compatible. Luckilly, vmware has a free player that works pretty well for my basic needs.\u003c/p\u003e","title":"Workaround for conflict between gnome-shell and vmware-player"},{"content":"Follow-up to my post about using Maildirs with Ubuntu\u0026rsquo;s messaging menu.\nBrad, the original author, has indicated that he considered his code to be essentially WTFPL. I feel the same about my bits, and have added a COPYING file specifically indicating this license instead of the previous GPL3 license.\nI have made some other changes since that last version:\nThe configuration values (frequency, directories) are now loaded from separate files. Default values are in /etc/maildir-indicator.conf, user-specific in $XDG_CONFIG_HOME/maildir-indicator.conf (or more likely, $HOME/.config/maildir-indicator.conf). This allows it to be installed system-wide and have users configure their own notifications. The .desktop file has been cleaned up, and is now advertising \u0026ldquo;Maildir Indicator\u0026rdquo; instead of mutt. This is what will be read in the menu as well (though of course, clicking it runs whatever mail agent you want). Cleaned up some debug messages to make the output a little easier to read Changed \u0026lsquo;path\u0026rsquo; to no longer have a trailing slash. I\u0026rsquo;m not sure what happens if you have a trailing slash on your paths, I didn\u0026rsquo;t test it. The program can still be used by grabbing the source from the Maildir-Indicator repository on Gitorious and running ./maildir-indicator in the directory you get. However, it also supports being installed system-wide, which can be done easily by installing it from the maildir-indicator PPA. Installing from the PPA will also set it to run automatically at login.\nTo use per-user configs, just copy the basic config to your .config directory, and edit it as needed:\ncp /etc/maildir-indicator.conf $HOME/.config/maildir-indicator.conf gedit $HOME/.config/maildir-indicator.conf One minor issue is that it has no way to see that your config has changed. You need to kill the running version, and start a new one. I didn\u0026rsquo;t worry about it since most people manually setting up maildir notifications are probably able to do that.\nIn the mean time, it seems that Brad has been busy as well! He\u0026rsquo;s taken the version from my previous post and updated it to provide much better notifications. It now reads the message and puts the sender and subject lines in the notification bubble! Now when a message arrives you can decide whether to read it now or later without opening your mail.\nThere is a bit of a logic bug with how my version handled the notifications: If the number of new messages is the same or less than the last notification, then it doesn\u0026rsquo;t re-notify. If you were notified of two messages, which you read, and a new one arrives before the program checked for new mail again, it won\u0026rsquo;t notify you about the new message (though the icon stays lit up). It assumes you only read one of the messages. To solve this, we would need to actually check out each file \u0026ndash; not in-depth, just the filename is good. Brad\u0026rsquo;s update goes beyond that. I\u0026rsquo;m not sure if he\u0026rsquo;s filtering out already-notified messages or not, but most of the pieces are there in his new version.\nThe bad news is my packaging work and his notification work happened in parallel. I was away for a week and didn\u0026rsquo;t see his updates until now that I\u0026rsquo;m posting this. I\u0026rsquo;ll merge his notification code tomorrow and release an 0.4 package. For reference, I\u0026rsquo;ve imported his version in my git repository under a branch called \u0026lsquo;brad\u0026rsquo; if you want to compare the differences (the merge should be fairly strait-forward).\nI couldn\u0026rsquo;t have said it better: \u0026ldquo;This is why I love Free/Open Source Software\u0026rdquo;.\nUPDATE 20110812: I\u0026rsquo;ve merged Brad\u0026rsquo;s detailed notification changes (with a few changes), and made a few other fixes (Thanks Nick for pointing out an issue in the comments).\n","permalink":"/posts/update-for-maildir-indicator/","summary":"\u003cp\u003eFollow-up to my post about using \u003ca href=\"/posts/maildir-with-ubuntus-messaging-menu\"\u003eMaildirs with Ubuntu\u0026rsquo;s messaging menu\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eBrad, the original author, has indicated that he considered his code to be essentially WTFPL. I feel the same about my bits,  and have added a COPYING file specifically indicating this license instead of the previous GPL3 license.\u003c/p\u003e\n\u003cp\u003eI have made some other changes since that last version:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eThe configuration values (frequency, directories) are now loaded from separate files. Default values are in /etc/maildir-indicator.conf, user-specific in $XDG_CONFIG_HOME/maildir-indicator.conf (or more likely, $HOME/.config/maildir-indicator.conf). This allows it to be installed system-wide and have users configure their own notifications.\u003c/li\u003e\n\u003cli\u003eThe .desktop file has been cleaned up, and is now advertising \u0026ldquo;Maildir Indicator\u0026rdquo; instead of mutt. This is what will be read in the menu as well (though of course, clicking it runs whatever mail agent you want).\u003c/li\u003e\n\u003cli\u003eCleaned up some debug messages to make the output a little easier to read\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eChanged \u0026lsquo;path\u0026rsquo; to no longer have a trailing slash\u003c/strong\u003e. I\u0026rsquo;m not sure what happens if you have a trailing slash on your paths, I didn\u0026rsquo;t test it.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe program can still be used by grabbing the source from the \u003ca href=\"https://gitorious.org/chrisirwin-utils/maildir-indicator/\"\u003eMaildir-Indicator repository on Gitorious\u003c/a\u003e and running \u003ccode\u003e./maildir-indicator\u003c/code\u003e in the directory you get. However, it also supports being installed system-wide, which can be done easily by installing it from the \u003ca href=\"https://launchpad.net/~chrisirwin/+archive/maildir-indicator\"\u003emaildir-indicator PPA\u003c/a\u003e. Installing from the PPA will also set it to run automatically at login.\u003c/p\u003e","title":"Update for maildir-indicator"},{"content":"I\u0026rsquo;ve created a Firefox addon collection. This is mostly for my own use (Since firefox can\u0026rsquo;t sync addons, a single list of addons I use helps bring up machines), but I thought I\u0026rsquo;d throw it out there if anybody else was curious.\n[Chris\u0026rsquo; Addons](https://addons.mozilla.org/en-US/firefox/collections/chrisirwin/chris-a/ Chris\u0026rsquo; Addons collection at addons.mozilla.org)\n","permalink":"/posts/my-own-firefox-addon-collection/","summary":"\u003cp\u003eI\u0026rsquo;ve created a Firefox addon collection. This is mostly for my own use (Since firefox can\u0026rsquo;t sync addons, a single list of addons I use helps bring up machines), but I thought I\u0026rsquo;d throw it out there if anybody else was curious.\u003c/p\u003e\n\u003cp\u003e[Chris\u0026rsquo; Addons](\u003ca href=\"https://addons.mozilla.org/en-US/firefox/collections/chrisirwin/chris-a/\"\u003ehttps://addons.mozilla.org/en-US/firefox/collections/chrisirwin/chris-a/\u003c/a\u003e Chris\u0026rsquo; Addons collection at addons.mozilla.org)\u003c/p\u003e","title":"My own Firefox Addon Collection"},{"content":"I was pretty sure I wasn\u0026rsquo;t the only mutt user who missed having mail notifications in the indicator area. From a quick google search, it turns out I was in good company. Even better, a fellow by the name of Brad Mont actually wrote a basic python script to create an appindicator that would monitor a maildir. It was about 90% of what I was looking for.\nUnfortunately, I didn\u0026rsquo;t want another icon crowding the panel, so I adapted his script to use the messaging menu, so it shows up with chat (and any other communications programs). I made a few other changes along the way (notifications). Thanks to the folks who wrote gm-notify for being such an easy to read example of both notification and the messaging menu.\nMy version is available as maildir-indicator on gitorious. Hopefully it will be of use to others as well.\n","permalink":"/posts/maildir-with-ubuntus-messaging-menu/","summary":"\u003cp\u003eI was pretty sure I wasn\u0026rsquo;t the only mutt user who missed having mail notifications in the indicator area. From a quick google search, it turns out I was in good company. Even better, a fellow by the name of \u003ca href=\"http://bradmont.net/\"\u003eBrad Mont\u003c/a\u003e actually wrote a basic python script to create an \u003ca href=\"http://bradmont.net/post/unity-indicator-applet-maildir-check\"\u003eappindicator that would monitor a maildir\u003c/a\u003e. It was about 90% of what I was looking for.\u003c/p\u003e\n\u003cp\u003eUnfortunately, I didn\u0026rsquo;t want another icon crowding the panel, so I adapted his script to use the messaging menu, so it shows up with chat (and any other communications programs). I made a few other changes along the way (notifications). Thanks to the folks who wrote gm-notify for being such an easy to read example of both notification and the messaging menu.\u003c/p\u003e","title":"Maildir with Ubuntu's Messaging Menu"},{"content":"Assuming you\u0026rsquo;re doing a Google Image Search for a specific unicomp keyboard.\nZOMG!\n","permalink":"/posts/im-internet-famous/","summary":"\u003cp\u003eAssuming you\u0026rsquo;re doing a Google Image Search for a specific unicomp keyboard.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"\u0026ldquo;screenshot\u0026rdquo;\" loading=\"lazy\" src=\"/posts/im-internet-famous/screenshot.png\"\u003e\u003c/p\u003e\n\u003cp\u003eZOMG!\u003c/p\u003e","title":"I'm Internet Famous!"},{"content":"As well as my previously mentioned Unicomp keyboard, I also decided to purchase a Logitech Trackman Wheel Optical. It\u0026rsquo;s basically a thumb-oriented trackball that resembles a standard mouse (with regards to clicking, scrolling, etc).\nAdjusting to using the trackball was extremely easy. I have one at work and one at home. I haven\u0026rsquo;t been able to use it for games yet, so I\u0026rsquo;ve still got a second mouse around for that (more on that in a bit…)\nI found the sensitivity to be a little low for my liking. There is a driver for Windows that allows you to adjust the sensitivity and acceleration of this mouse, but this requires running their software. Xorg has this built-in, but that level of device-specific tuning is not exposed easily. After playing around with xinput and xset, I\u0026rsquo;ve settled on some xorg.conf.d rules that accomplish the same thing:\n# /usr/share/X11/xorg.conf.d/75-trackman.conf # Configuration settings for Logitech Trackball Section \u0026quot;InputClass\u0026quot; Identifier \u0026quot;Logitech Trackball\u0026quot; MatchIsPointer \u0026quot;on\u0026quot; MatchDevicePath \u0026quot;/dev/input/event*\u0026quot; MatchProduct \u0026quot;Logitech Trackball\u0026quot; Driver \u0026quot;evdev\u0026quot; Option \u0026quot;MinSpeed\u0026quot; \u0026quot;2.00\u0026quot; Option \u0026quot;MaxSpeed\u0026quot; \u0026quot;3.00\u0026quot; #Option \u0026quot;AccelFactor\u0026quot; \u0026quot;0.40\u0026quot; Option \u0026quot;AccelerationNumerator\u0026quot; \u0026quot;5\u0026quot; Option \u0026quot;AccelerationDenominator\u0026quot; \u0026quot;1\u0026quot; EndSection The biggest issue I experienced was in Windows 7. The software for this trackball, and the software for my Logitech G9 mouse are different. They have the same name, but they do not seem to be able to configure each other, requiring both the be installed and running. That said, they don\u0026rsquo;t seem to interfere with each other either. I find it somewhat sloppy, but I don\u0026rsquo;t use Windows for any serious work anyway so it hasn\u0026rsquo;t become an issue.\nThe ball doesn\u0026rsquo;t sit on rollers like old mechanical trackball mice, but on three slick pads. The ball movement is tracked optically. The pads seem to collect a fair amount of gunk. I clean it out every few days, though I haven\u0026rsquo;t noticed any ill effects yet (other than needing to wash my hands more often).\nPros Don\u0026rsquo;t have to swing your arm all over I find it quite relaxing to use Can put it on your leg and lean back in your chair Cons Software required to configure for Windows Windows software doesn\u0026rsquo;t work for other Logitech mice Pads holding ball get dirty rather quick. Your mileage may vary. Conclusion For \u0026ldquo;desktop\u0026rdquo; use, it sure beats a mouse. It is definately worth $30, without question.\n","permalink":"/posts/logitech-trackman-wheel-optical/","summary":"\u003cp\u003eAs well as my previously mentioned \u003ca href=\"/posts/unicomp-customizer-104\"\u003eUnicomp keyboard\u003c/a\u003e, I also decided to purchase a \u003ca href=\"http://www.amazon.com/Logitech-Trackman-Wheel-Optical-Silver/dp/B00005NIMJ\"\u003eLogitech Trackman Wheel Optical\u003c/a\u003e.  It\u0026rsquo;s basically a thumb-oriented trackball that resembles a standard mouse (with regards to clicking, scrolling, etc).\u003c/p\u003e\n\u003cp\u003eAdjusting to using the trackball was extremely easy. I have one at work and one at home. I haven\u0026rsquo;t been able to use it for games yet, so I\u0026rsquo;ve still got a second mouse around for that (more on that in a bit…)\u003c/p\u003e","title":"Review: Logitech Trackman Wheel Optical"},{"content":"I\u0026rsquo;ve recently purchased a Unicomp Customizer 104. It is essentially a modern version of the old IBM Model M mechanical keyboards. I do a fair amount of typing, so I thought it would be worthwhile to invest in a better keyboard.\nI\u0026rsquo;ve decided not to post photos since there seems to be no shortage of those on teh interwebs already.\nThe feel is very nice. It reminds me of the Model M I used to have, but unfortunately, that model is long gone and I can\u0026rsquo;t do a direct comparison. The keys are very loud, which was expected from this type of mechanism (Buckling Spring). I expected the actual mechanism to buckle roughly 50-75% of the way down so that an experienced typist can stop their finger motion before they hit the backplate. It appears that these keys require almost 95% motion, at least by my rough estimate. I consider it impossible to type without moving the key completely down. Unfortunate, but no worse than any other keyboard.\nThe keyboard is physically large. There is a slimmer version, but I opted for the large customizer for nostalgia.\nI rather like it. The folks at Unicomp were very accommodating. I originally ordered a different board, then changed my order. They corrected it and shipped it right away. Unfortunately the billing still went through for the original (more expensive) keyboard, but the difference was refunded right away when I contacted them.\nPros Very clicky Visually appealing keyboard Fun to type on Loud Cons Tactile feedback not as awesome as expected. A little pricey Loud Conclusion I quite like this keyboard. I was a little worried about paying USD$70, plus shipping, plus fees (Import via Fedex into Canada). I consider it a rather expensive keyboard, but I definitely enjoy typing on it more than the thinkpad keyboard, or the Microsoft X6 board I was using before.\n","permalink":"/posts/unicomp-customizer-104/","summary":"\u003cp\u003eI\u0026rsquo;ve recently purchased a \u003ca href=\"http://pckeyboards.stores.yahoo.net/customizer.html\"\u003eUnicomp Customizer 104\u003c/a\u003e. It is essentially a modern version of the old IBM Model M mechanical keyboards. I do a fair amount of typing, so I thought it would be worthwhile to invest in a better keyboard.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve decided not to post photos since there seems to be no shortage of those on \u003ca href=\"http://www.google.ca/images?q=unicomp+customizer+104\"\u003eteh interwebs\u003c/a\u003e already.\u003c/p\u003e\n\u003cp\u003eThe feel is very nice. It reminds me of the Model M I used to have, but unfortunately, that model is long gone and I can\u0026rsquo;t do a direct comparison. The keys are very loud, which was expected from this type of mechanism (\u003ca href=\"http://en.wikipedia.org/wiki/Buckling_spring\"\u003eBuckling Spring\u003c/a\u003e). I expected the actual mechanism to buckle roughly 50-75% of the way down so that an experienced typist can stop their finger motion before they hit the backplate. It appears that these keys require almost 95% motion, at least by my rough estimate. I consider it impossible to type without moving the key completely down. Unfortunate, but no worse than any other keyboard.\u003c/p\u003e","title":"Review: Unicomp Customizer 104"},{"content":"I\u0026rsquo;ve tried (and failed) to use many different pieces of software designed to manage todo lists. The main reasons I\u0026rsquo;ve failed is because the software either has a high learning curve, bad documentation, or it is cumbersome to use.\nI\u0026rsquo;ve finally struck gold with devtodo. Out of the box, it is almost perfect, but there are a few little issues:\nIt expects .todo in the current directory It has no ability to track what you are working on. I\u0026rsquo;ve managed to work around both of those with some extra functions in my .bash_profile.\nWhere is my .todo? I wanted a git-like approach to my todo list. If there is not one in the current directory, check the parent, the grandparent, etc. Eventually, fall-back to \u0026lsquo;Global\u0026rsquo; todo list in ${HOME}.\nNote: This depends on the rfind function in my previous post.\nFirst, create a function to set the TODO_FILE variable. We\u0026rsquo;re going to set TASK_FILE here as well, this is referenced further down the page.\nset_nearest_todo_file() { ntdf_file=$( rfind -name .todo -type f ) if [[ -z \u0026quot;${ntdf_file}\u0026quot; ]]; then ntdf_file=\u0026quot;${HOME}/.todo\u0026quot; fi TODO_FILE=\u0026quot;${ntdf_file}\u0026quot; } set_nearest_task_file() { TASK_FILE=\u0026quot;${TODO_FILE}-current\u0026quot; } Now, we need to call these on cd, pushd, and popd, as well as when the shell is sourced.\ncd() { if builtin cd \u0026quot;$@\u0026quot;; then set_nearest_todo_file set_nearest_task_file fi } pushd() { if builtin pushd \u0026quot;$@\u0026quot;; then set_nearest_todo_file set_nearest_task_file fi } popd() { if builtin popd \u0026quot;$@\u0026quot;; then set_nearest_todo_file set_nearest_task_file fi } set_nearest_todo_file set_nearest_task_file Next, lets make an alias for td to call devtodo with our desired todo_file\ntd() { # Don't print DB notice for .todo in current dir if [[ \u0026quot;${TODO_FILE}\u0026quot; != \u0026quot;./.todo\u0026quot; ]]; then # Output DB notice to stderr. # so we don't mess with output parsers echo \u0026quot;Using database ${TODO_FILE}\u0026quot; 1\u0026gt;\u0026amp;2 fi # Specify found DB devtodo --database ${TODO_FILE} $@ } Now, when we call td, we will be referencing the nearest TODO_FILE, either in the current directory, one of it\u0026rsquo;s parents, or falling back to ${HOME}.\nWhat am I working on? This was simple enough. devtodo has aliases for todo (devtodo), tda (todo \u0026ndash;add) and tdd (todo \u0026ndash;done). We also added td above. I\u0026rsquo;m going to hijack tdd, and add a new \u0026rsquo;t\u0026rsquo; command to set a current task.\nWe already added the neccessary parts to get TASK_FILE above. Now we\u0026rsquo;re just going to create a function to set a task in that file.\nt() { if [[ \u0026quot;$*\u0026quot; == \u0026quot;\u0026quot; ]] ; then if [[ -f \u0026quot;${TASK_FILE}\u0026quot; ]]; then cat ${TASK_FILE} fi else td $@ | tee ${TASK_FILE} fi } You\u0026rsquo;ll notice that if we call t with a parameter, such as t 6, it run td 6, instruct td to show task 6, and copy that output to your TASK_FILE.\nIf we simply call t, it will cat out our TASK_FILE to the screen.\nI\u0026rsquo;ve also hijacked tdd, the devtodo \u0026lsquo;done\u0026rsquo; alias, to clear my TASK_FILE as well.\ntdd() { /usr/bin/tdd $@ rm ${TASK_FILE} } My workflow I find this very easy to manage. td to review my list, t # to select my active task, and t while I\u0026rsquo;m working to be reminded what I\u0026rsquo;m doing. The task number is still in the output of t. When I\u0026rsquo;m done, tdd # marks that task as complete, removes TASK_FILE, and I\u0026rsquo;m free to start another task.\n","permalink":"/posts/using-devtodo-with-multiple-projects/","summary":"\u003cp\u003eI\u0026rsquo;ve tried (and failed) to use many different pieces of software\ndesigned to manage todo lists. The main reasons I\u0026rsquo;ve failed is because\nthe software either has a high learning curve, bad documentation, or it\nis cumbersome to use.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve finally struck gold with devtodo. Out of the box, it is almost\nperfect, but there are a few little issues:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt expects .todo in the current directory\u003c/li\u003e\n\u003cli\u003eIt has no ability to track what you are working on.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI\u0026rsquo;ve managed to work around both of those with some extra functions in\nmy .bash_profile.\u003c/p\u003e","title":"Using devtodo with multiple projects"},{"content":"I recently had need to do a reverse find, and couldn\u0026rsquo;t discover any programs that offer this functionality. I decided to work around the issue using a bash function, loops, and find.\nThis function can be stuffed into your ~/.bash_profile, and referenced wherever you need it.\nrfind() { rfind_path=\u0026amp;quot;${PWD}\u0026amp;quot; while [[ \u0026amp;quot;${rfind_path}\u0026amp;quot; != \u0026amp;quot;/\u0026amp;quot; ]]; do rfind_search_paths=\u0026amp;quot;${rfind_search_paths} ${rfind_path}\u0026amp;quot; rfind_path=$(dirname \u0026amp;quot;${rfind_path}\u0026amp;quot;) done find ${rfind_search_paths} / -maxdepth 1 $@ -print -quit } As an example, let\u0026rsquo;s say I\u0026rsquo;m in /home/user/docs, and I execute rfind somefile.txt. rfind will actually build a command and execute the following:\nfind /home/user/docs /home/user /home / -maxdepth 1 somefile.txt -print -quit This will find the nearest parent directory containing a file called somefile.txt, and quit after printing the first (nearest) match.\nEdit: Updated rfind to use a much simpler dirname method.\n","permalink":"/posts/reverse-find/","summary":"\u003cp\u003eI recently had need to do a reverse find, and couldn\u0026rsquo;t discover any programs that offer this functionality. I decided to work around the issue using a bash function, loops, and \u003ccode\u003efind\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eThis function can be stuffed into your ~/.bash_profile, and referenced wherever you need it.\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003erfind()\n{\n    rfind_path=\u0026amp;quot;${PWD}\u0026amp;quot;\n    while [[ \u0026amp;quot;${rfind_path}\u0026amp;quot; != \u0026amp;quot;/\u0026amp;quot; ]]; do\n        rfind_search_paths=\u0026amp;quot;${rfind_search_paths} ${rfind_path}\u0026amp;quot;\n        rfind_path=$(dirname \u0026amp;quot;${rfind_path}\u0026amp;quot;)\n    done\n    \n    find ${rfind_search_paths} / -maxdepth 1 $@ -print -quit\n}\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eAs an example, let\u0026rsquo;s say I\u0026rsquo;m in /home/user/docs, and I execute \u003ccode\u003erfind somefile.txt\u003c/code\u003e. rfind will actually build a command and execute the following:\u003c/p\u003e","title":"Reverse Find"},{"content":"Okay, so the ipad is just a big ipod touch, and filling the \u0026ldquo;device too large to always have on you but too limited to do lots with\u0026rdquo; between a handy with-you smartphone and a real computer.\nIf it was an accessory to a computer, THEN it would be interesting. Put it on the dock and it acts like a second screen. If you\u0026rsquo;re reading a PDF or web page, you can send it to the ipad for display, grab the ipad and continue having your reference material up. How about an example:\nI look up \u0026ldquo;How to do XYZ on some obscure server issue\u0026rdquo; on my computer. I do this on my computer because it takes a long time to find what I need. I read through until I find the helpful bit on page 322 of a 700 page pdf. I send that to my padd and walk to the server room where I can follow the instructions Note there was no step for manually copying a file, bookmarking, manually opening on the padd, remembering I want p322, etc. Just drag send it to the padd (whether it\u0026rsquo;s a drag, a click, whatever). For inspiration, see interesting UI interaction done on Science Fiction. Star Trek has had accessory \u0026ldquo;padd\u0026rdquo; devices that acted this way. Minority Report had this as well (which was mostly un-noticed next to the gesture UI everybody did happen to notice).\nUntil that device comes out, I see no need for a cell phone that can\u0026rsquo;t make calls, or a computer that can\u0026rsquo;t run software.\n","permalink":"/posts/the-ipad-letdown/","summary":"\u003cp\u003eOkay, so the ipad is just a big ipod touch, and filling the \u0026ldquo;device too large to always have on you but too limited to do lots with\u0026rdquo; between a handy with-you smartphone and a real computer.\u003c/p\u003e\n\u003cp\u003eIf it was an accessory to a computer, THEN it would be interesting. Put it on the dock and it acts like a second screen. If you\u0026rsquo;re reading a PDF or web page, you can send it to the ipad for display, grab the ipad and continue having your reference material up. How about an example:\u003c/p\u003e","title":"the iPad letdown"},{"content":"I\u0026rsquo;ve linked my blog to [Twitter](http://twitter.com Twitter), [Identi.ca](http://identi.ca identi.ca), and [Facebook](http://facebook.com Facebook) via [ping.fm](http://ping.fm ping dot fm) via [CR Post2Pingfm](http://wordpress.org/extend/plugins/cr-post2pingfm/ CR Post to Ping fm) [Twitterfeed](http://twitterfeed.com/ Twitter Feed dot com). Whew.\nHopefully this means that I will update my blog more often as there is a slight, remote chance that somebody might actually see it.\nI\u0026rsquo;ve got two LUG presentations coming up: DIY debian packaging for [LOLUG](http://groups.google.ca/group/lolug London Ontario Linux Users Group) and KVM+libvirt for [KWLUG](http://kwlug.org/ Kitchener-Waterloo Linux Users Group). I\u0026rsquo;ll post slides when I present.\nUpdate: I\u0026rsquo;ve decided to use twitterfeed instead of the wordpress plugin I was using. It manages to post a little nicer to facebook. The only down-side is it is a timed rss-pull mechanism instead of a nice push, but I\u0026rsquo;ll live.\n","permalink":"/posts/for-twits-and-bookfacers/","summary":"\u003cp\u003eI\u0026rsquo;ve linked my blog to [Twitter](\u003ca href=\"http://twitter.com\"\u003ehttp://twitter.com\u003c/a\u003e Twitter), [Identi.ca](\u003ca href=\"http://identi.ca\"\u003ehttp://identi.ca\u003c/a\u003e identi.ca), and [Facebook](\u003ca href=\"http://facebook.com\"\u003ehttp://facebook.com\u003c/a\u003e Facebook) via [ping.fm](\u003ca href=\"http://ping.fm\"\u003ehttp://ping.fm\u003c/a\u003e ping dot fm) via [CR Post2Pingfm](\u003ca href=\"http://wordpress.org/extend/plugins/cr-post2pingfm/\"\u003ehttp://wordpress.org/extend/plugins/cr-post2pingfm/\u003c/a\u003e CR Post to Ping fm) [Twitterfeed](\u003ca href=\"http://twitterfeed.com/\"\u003ehttp://twitterfeed.com/\u003c/a\u003e Twitter Feed dot com). Whew.\u003c/p\u003e\n\u003cp\u003eHopefully this means that I will update my blog more often as there is a slight, remote chance that somebody might actually see it.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve got two LUG presentations coming up: DIY debian packaging for [LOLUG](\u003ca href=\"http://groups.google.ca/group/lolug\"\u003ehttp://groups.google.ca/group/lolug\u003c/a\u003e London Ontario Linux Users Group) and KVM+libvirt for [KWLUG](\u003ca href=\"http://kwlug.org/\"\u003ehttp://kwlug.org/\u003c/a\u003e Kitchener-Waterloo Linux Users Group). I\u0026rsquo;ll post slides when I present.\u003c/p\u003e","title":"For Twits and bookfacers"},{"content":"I\u0026rsquo;ve created a new project called imagecompare. The purpose is to detect and merge duplicate images, particularly where EXIF information may differ.\nThere is more information on the project page\n","permalink":"/posts/imagecompare/","summary":"\u003cp\u003eI\u0026rsquo;ve created a new project called imagecompare. The purpose is to detect and merge duplicate images, particularly where EXIF information may differ.\u003c/p\u003e\n\u003cp\u003eThere is more information on \u003ca href=\"/blog/projects/imagecompare/\"\u003ethe project page\u003c/a\u003e\u003c/p\u003e","title":"imagecompare"},{"content":"I\u0026rsquo;ve recently switched over to using git for revision control for my personal projects. This has made doing packaging and development substantially easier.\nI also decided to use git for my resume. I\u0026rsquo;ve redone my resume in LyX (a LaTeX front end)\n","permalink":"/posts/git-316/","summary":"\u003cp\u003eI\u0026rsquo;ve recently switched over to using git for revision control for my personal projects. This has made doing packaging and development substantially easier.\u003c/p\u003e\n\u003cp\u003eI also decided to use git for my resume. I\u0026rsquo;ve redone my resume in LyX (a LaTeX front end)\u003c/p\u003e","title":"Git"},{"content":"I have a [PPA on Launchpad.net](https://edge.launchpad.net/~chrisirwin/+archive/ppa Chris Irwin\u0026rsquo;s PPA) now. I\u0026rsquo;ve found it substantially easier than managing my own repository. Apparently some other folks use it, so I will be posting here when I do notable updates and additions. Here is a breakdown of what is on there now:\ncwi-meta and the cwi-* packages it builds. These are packages that provide PPA repository information and package dependencies for me setting up new (virtual) machines. One apt-get and I have my expected environment. libvirt, virt-manager, and associated packages. I\u0026rsquo;m keeping these up to date for Intrepid as I am using them with kvm/qemu. cwi-vm-builder. A wrapper around ubuntu-vm-builder. When compined with an apt proxy/cache, it can bootstrap a VM in about 2 minutes. This is my standard vm template so I can build a new VM in a single command. task \u0026amp; yagtd. Two unrelated gtd-style task managers. I reccommend trying them out. If you use my PPA, feel free to contact me with suggestions and issues. I can\u0026rsquo;t fix anything I don\u0026rsquo;t know about. Also, launchpad does not give any statistics on PPA useage, or download history. It would be nice to get a rough count of the number of users using my packages.\n","permalink":"/posts/my-ubuntu-ppa-packages/","summary":"\u003cp\u003eI have a [PPA on Launchpad.net](\u003ca href=\"https://edge.launchpad.net/~chrisirwin/+archive/ppa\"\u003ehttps://edge.launchpad.net/~chrisirwin/+archive/ppa\u003c/a\u003e Chris Irwin\u0026rsquo;s PPA) now. I\u0026rsquo;ve found it substantially easier than managing my own repository. Apparently some other folks use it, so I will be posting here when I do notable updates and additions. Here is a breakdown of what is on there now:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ecwi-meta\u003c/strong\u003e and the cwi-* packages it builds. These are packages that provide PPA repository information and package dependencies for me setting up new (virtual) machines. One apt-get and I have my expected environment.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003elibvirt\u003c/strong\u003e, \u003cstrong\u003evirt-manager\u003c/strong\u003e, and associated packages. I\u0026rsquo;m keeping these up to date for Intrepid as I am using them with kvm/qemu.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ecwi-vm-builder\u003c/strong\u003e. A wrapper around ubuntu-vm-builder. When compined with an apt proxy/cache, it can bootstrap a VM in about 2 minutes. This is my standard vm template so I can build a new VM in a single command.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003etask\u003c/strong\u003e \u0026amp; \u003cstrong\u003eyagtd\u003c/strong\u003e. Two unrelated gtd-style task managers. I reccommend trying them out.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf you use my PPA, feel free to contact me with suggestions and issues. I can\u0026rsquo;t fix anything I don\u0026rsquo;t know about. Also, launchpad does not give any statistics on PPA useage, or download history. It would be nice to get a rough count of the number of users using my packages.\u003c/p\u003e","title":"My Ubuntu PPA packages"},{"content":"I have imported content from my previous Wordpress blog (It was MIA for a bit). Pretty much everything older than this post is obsolete.\n","permalink":"/posts/bulk-import/","summary":"\u003cp\u003eI have imported content from my previous Wordpress blog (It was MIA for a bit). Pretty much everything older than this post is obsolete.\u003c/p\u003e","title":"Bulk Import"},{"content":"Here are some interesting links to keep handy. I do not think these are easy enough to come across, so I\u0026rsquo;m linking to them from here.\nHow to get windows to use a hardware clock set to GMT. You know, like the rest of the world.\nHow to grab the i-sight firmware for use with Linux. Note that the kernel in Ubuntu Feisty already has the driver part.\nSome Ubuntu email server tips.\n","permalink":"/posts/things-to-remember-63/","summary":"\u003cp\u003eHere are some interesting links to keep handy. I do not think these are easy enough to come across, so I\u0026rsquo;m linking to them from here.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://discussions.apple.com/message.jspa?messageID=2378251#2378251\"\u003eHow to get windows to use a hardware clock set to GMT\u003c/a\u003e. You know, like the rest of the world.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://ubuntuforums.org/showthread.php?t=225621\"\u003eHow to grab the i-sight firmware for use with Linux\u003c/a\u003e. Note that the kernel in Ubuntu Feisty already has the driver part.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ca href=\"http://www.murrayc.com/blog/permalink/2006/09/19/installing-an-email-server-on-ubuntu/\"\u003eSome Ubuntu email server tips\u003c/a\u003e.\u003c/p\u003e","title":"Things to remember"},{"content":"I have various pieces of computer equipment that I am looking to get rid of. I would much prefer to give them to somebody who can put them to a useful purpose rather than throw them out. Please contact me if you are interested.\nI\u0026rsquo;m listing all components separately. While several are currently attached (motherboards + cpus) I am willing to divide up this stuff however necessary.\nFree Stuff Motherboards + CPUs 2 - 1.2GHz AMD Athlon (Thunderbird) CPUs Need heatsinks, probably around $20 nowadays.\n1 - 1.0GHz AMD Duron CPU 2 - MSI Motherboards 1 - FIC Motherboard Optical Drives 1 - 52x32x52 CD ReWriter/16x DVD-ROM 1 - 48x16x48 CD ReWriter 1 - 4x4x?x IDE CD ReWriter 1 - ?x DVD DVD-ROM (unsure of speed) 1 - 6x IDE CD-ROM 1 - 24x CD-ROM ##Not Free Stuff\n1 - Apple iBook 12\u0026quot; 1.07GHz G4 768 MB Ram 40 GB Hard Disk - I\u0026rsquo;ll throw in the original 30GB Hard Disk, but it is not easily changeable. 32MB ATI Radeon 9200 1024x768 display Fancy Slot-loading CD ReWriter/DVD-Rom Airport Express (802.11b/g) Brand-new battery - Just had the battery replaced through an Apple product recall, so this battery has full capacity. OS X 10.4 + Original 10.3 Disks Fancy metal briefcase (It is actually multi purpose, I packed it with cut-out foam) Runs Linux extremely well, but it currently has only OS X on it. I\u0026rsquo;d be willing to install Ubuntu on it if desired. ","permalink":"/posts/some-free-to-a-good-home-stuff/","summary":"\u003cp\u003eI have various pieces of computer equipment that I am looking to get rid of. I would much prefer to give them to somebody who can put them to a useful purpose rather than throw them out. Please contact me if you are interested.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m listing all components separately. While several are currently attached (motherboards + cpus) I am willing to divide up this stuff however necessary.\u003c/p\u003e\n\u003ch2 id=\"free-stuff\"\u003eFree Stuff\u003c/h2\u003e\n\u003ch3 id=\"motherboards--cpus\"\u003eMotherboards + CPUs\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e2 - 1.2GHz AMD Athlon (Thunderbird) CPUs\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eNeed heatsinks, probably around $20 nowadays.\u003c/p\u003e","title":"Some \u0026quot;Free to a good home\u0026quot; stuff"},{"content":"I was just building some statistics of users, and apparently there are 82 unique IPs that access my package repository. This excludes my workstations. I had no idea anybody else was even using them, so \u0026ldquo;Rock On\u0026rdquo; as it were.\nI\u0026rsquo;ll be creating an RSS feed soon to keep users appraised of future changes to the repository. Also, I apologize for the reorganization that happened a few weeks ago, as it probably had caused some headaches.\nMost of the packages are config/build changes (Metacity to enable the compositor, mail-notification to enable ssl), some are backports (libcm7 for Metacity dependency), and some are modifications.\nGaim now has an additional package, gaim-icons-official. This breaks the main UI icons into their own package that is replaceable via the packaging system. gaim-icons-gperfection2 is already in my repository, and I\u0026rsquo;ll be making a gaim-icons-tango as soon as I have a few spare minutes. Not as cool as having switchable icon theme support in gaim itself, but good nevertheless.\nJust a note that I do take requests for packages and backports, and will be putting an effort (now that I\u0026rsquo;ve gotten the hang of the whole Debian build stuff) to get changes into Ubuntu or upstream (if possible/necessary).\n","permalink":"/posts/ubuntu-packages/","summary":"\u003cp\u003eI was just building some statistics of users, and apparently there are 82 unique IPs that access my \u003ca href=\"http://packages.cidesign.ca\"\u003epackage repository\u003c/a\u003e. This excludes my workstations. I had no idea anybody else was even using them, so \u0026ldquo;Rock On\u0026rdquo; as it were.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ll be creating an RSS feed soon to keep users appraised of future changes to the repository. Also, I apologize for the reorganization that happened a few weeks ago, as it probably had caused some headaches.\u003c/p\u003e","title":"Ubuntu packages"},{"content":"There is a rather interesting documentary about Hugo Chavez of Venezuala.\nThe Revolution will not be Televised.\n","permalink":"/posts/the-revolution-will-not-be-televised-9/","summary":"\u003cp\u003eThere is a rather interesting documentary about Hugo Chavez of Venezuala.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://video.google.com/videoplay?docid=5832390545689805144\u0026amp;q=the+revolution+will+not+be+televised\"\u003eThe Revolution will not be Televised\u003c/a\u003e.\u003c/p\u003e","title":"The Revolution will not be Televised"},{"content":"[ zaphod ] ~ $ ping -c 1 yahoo.com PING yahoo.com (66.94.234.13) 56(84) bytes of data. 64 bytes from **fifaworldcup**.yahoo.net (66.94.234.13): icmp_seq=1 ttl=52 time=83.8 ms --- yahoo.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 83.859/83.859/83.859/0.000 ms ","permalink":"/posts/wtf-yahoo/","summary":"\u003cpre\u003e\u003ccode\u003e[ zaphod ] ~ $ ping -c 1 yahoo.com\nPING yahoo.com (66.94.234.13) 56(84) bytes of data.\n64 bytes from **fifaworldcup**.yahoo.net (66.94.234.13): icmp_seq=1 ttl=52 time=83.8 ms\n\n--- yahoo.com ping statistics ---\n1 packets transmitted, 1 received, 0% packet loss, time 0ms\nrtt min/avg/max/mdev = 83.859/83.859/83.859/0.000 ms\n\u003c/code\u003e\u003c/pre\u003e","title":"WTF Yahoo"},{"content":"So I sent my MacBook in for repairs. The fn key seems to have come right off, which is unfortunately not only a useful key, but perhaps the most heavily-trafficked key on the entire board (being a pre-requisite of such auspicious keys such as Page-Up, and Home). Luckily, the guy at the store thinks it may be ready tomorrow \u0026ndash; assuming they can diagnose the missing key and quickly order a replacement keyboard. I forgot to mention the wonky trackpad issue \u0026ndash; perhaps a service ticket for another day.\nTo compound issues, however, my brother\u0026rsquo;s monitor has gone fuzzy. It is under warranty for several more days (read: four more days), and we took it in last week. Futureshop, of course, sent the monitor (a near three-year-old 19-inch) away for repair. Me being the kind, considerate and modest person I am offered my brother my own monitor in exchange. \u0026ldquo;After all, I mainly use the laptop!\u0026rdquo;\nSo hopefully I am only computerless for only a single day\u0026hellip;\n\u0026hellip;Although technically I am not computerless, as I still have my desktop, my server, and the old PII dell I am setting up as an arcade/myth box\u0026hellip;\nSo hopefully I am only displayless for only a single day\u0026hellip;\n","permalink":"/posts/breaky-breaky/","summary":"\u003cp\u003eSo I sent my MacBook in for repairs. The fn key seems to have come right off, which is unfortunately not only a useful key, but perhaps the most heavily-trafficked key on the entire board (being a pre-requisite of such auspicious keys such as Page-Up, and Home). Luckily, the guy at the store thinks it may be ready tomorrow \u0026ndash; assuming they can diagnose the missing key and quickly order a replacement keyboard. I forgot to mention the wonky trackpad issue \u0026ndash; perhaps a service ticket for another day.\u003c/p\u003e","title":"Breaky Breaky"},{"content":"My old Phone sucked. It was a small Kyocera slider that had consistently horrible reception and call quality. I was able to opt for a hardware update today, and decided to jump for it. I got a Motorola e815.\nIt seems to have excellent call quality and reception, though I\u0026rsquo;ve only had it a few hours, so this may be rather subjective. The website linked above seems to indicate the phone does not have Java, but mine apparently does. This may be a bell mobility thing, however. I also seem to be able to transfer photos to and from the camera itself via the Apple Bluetooth tools, as well as use isync with it.\niSync setup was pretty simple.\nSelect \u0026ldquo;Set up Bluetooth Device\u0026rdquo; from the bluetooth menu (or go through System Preferences) Follow instructions Palm Tungsten T3\nThis also provided me with the kick required to actually get around to using my Palm again. For a while, palm sync on Linux sucked, and I never really used it with windows due to crappy application support. The procedure to get bluetooth iSync working was less than straight forward for both Apple and Palm\u0026rsquo;s respective sides.\nOn the Mac:\nInstall Palm Desktop Set up Hotsync Manager Ensure palm can sync Run iSync Select \u0026ldquo;Enable Palm OS Syncing\u0026rdquo; from the \u0026ldquo;Devices\u0026rdquo; menu Enable the iSync conduit in Hotsync Manager Configure in iSync On the palm side, it seems that after I try to add a bluetooth connection to the phone, it outright refuses to sync to the macbook, even after a reset. I am not sure why it does this, but I eventually had to settle for hard-resetting the device (The battery had run down anyway, so there was no data to worry about).\nI am not sure whether it is worthwhile to attempt another phone link\u0026hellip;\n","permalink":"/posts/motorola-e815-and-palm-t3-with-isync/","summary":"\u003cp\u003eMy old Phone sucked. It was a small Kyocera slider that had consistently horrible reception and call quality. I was able to opt for a hardware update today, and decided to jump for it. I got a \u003ca href=\"http://www.phonescoop.com/phones/phone.php?p=663\"\u003eMotorola e815\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eIt seems to have excellent call quality and reception, though I\u0026rsquo;ve only had it a few hours, so this may be rather subjective. The website linked above seems to indicate the phone does not have Java, but mine apparently does. This may be a bell mobility thing, however. I also seem to be able to transfer photos to and from the camera itself via the Apple Bluetooth tools, as well as use isync with it.\u003c/p\u003e","title":"Motorola e815 and Palm T3 with iSync"},{"content":"I\u0026rsquo;ve spent the time since my last post playing with encfs and libpam-encfs. It is actually rather easy to have working, and I haven\u0026rsquo;t noticed much of a slow-down for regular system use yet.\n","permalink":"/posts/libpam-encfs/","summary":"\u003cp\u003eI\u0026rsquo;ve spent the time since my last post playing with encfs and libpam-encfs. It is actually rather easy to have working, and I haven\u0026rsquo;t noticed much of a slow-down for regular system use yet.\u003c/p\u003e","title":"libpam-encfs"},{"content":"Playing around with PAM modules lately, I have discovered libpam-ssh and pam-keyring. Both require some mucking about in /etc/pam.d/, but provide some worthwhile benefits.\nlibpam-ssh provides automatic spawning of ssh-agent, and populating ssh keys as required. This is handier than the keychain package as you do not have to enter passwords for each key, or even at all other than the initial login. The downside, of course, is that your ssh key passphrases must be the same as your system password.\npam-keyring does the same thing for gnome\u0026rsquo;s keyring. Very handy if you use Network Manager for wifi.\nLater on, I plan to look at using pam-encfs to encrypt either all (or portions if all is not feasible) of my home directory without requiring further password entry during login or boot.\n","permalink":"/posts/pam-modules/","summary":"\u003cp\u003ePlaying around with PAM modules lately, I have discovered libpam-ssh and pam-keyring. Both require some mucking about in /etc/pam.d/, but provide some worthwhile benefits.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003elibpam-ssh\u003c/strong\u003e provides automatic spawning of ssh-agent, and populating ssh keys as required. This is handier than the keychain package as you do not have to enter passwords for each key, or even at all other than the initial login. The downside, of course, is that your ssh key passphrases must be the same as your system password.\u003c/p\u003e","title":"PAM Modules"},{"content":"I have changed packages.cidesign.ca again. It is now i386 only, as I do not really use ppc since I got the macbook, and I can not maintain it.\nI have added updated packages for dosbox 0.65, as well as scummvm 0.9.0. I have modified scummvm to install the modern theme, and to search a system-wide path for themes (/usr/share/scummvm/themes). It seems to work alright, and looks a whole lot nicer.\nI\u0026rsquo;ve started trying to sign packages, but for some reason it is still saying they are not authenticated. I\u0026rsquo;m not sure why, and will need to look into this more later.\n","permalink":"/posts/package-changes/","summary":"\u003cp\u003eI have changed packages.cidesign.ca again. It is now i386 only, as I do not really use ppc since I got the macbook, and I can not maintain it.\u003c/p\u003e\n\u003cp\u003eI have added updated packages for dosbox 0.65, as well as scummvm 0.9.0. I have modified scummvm to install the modern theme, and to search a system-wide path for themes (/usr/share/scummvm/themes). It seems to work alright, and looks a whole lot nicer.\u003c/p\u003e","title":"Package Changes"},{"content":"So I purchased a Macbook. I waited until I had it for a bit before posting about it. I\u0026rsquo;ve only had it for two days, but am really enjoying it thus far. I will post photos later.\nThis is not a review. There are enough of those already.\nThe screen is excellent, the keyboard only takes a few minutes to get used to, the touch-pad is very natural, and the heat issues are\u0026hellip; intermittent, and very dependent on ambient-temperature. During the day, my machine tends to idle anywhere from 50 - 60 Celsius. Night tends to drop a few degrees to 45 - 52.\nLoad is a very different matter. Temperature seems to peek at about 80 or so, when the fan kicks on full (the fan starts at low speed at around 65). After the fan runs at full speed, it brings the temperature down to about 70-72, where it seems to stay.\nToday I slipstreamed Service Pack 2 onto my Windows XP installation CD, as well as downloading the Ubuntu Dapper release CD (I had previously used pre-release builds to install). I did not get around to installing, however, due to some stability issues in Mac OS X.\nFirst of all, allow me to thank apple very much for the hardware diagnostic utility on the install disc! Holding \u0026lsquo;D\u0026rsquo; while booting with the install disc in the system brings up a utility that will do a complete hardware diagnostic. As it turns out, one of my 1GB SO-DIMMs is bad. After testing each piece (2x256 it shipped with, and 2x1GB I replaced them with) in each slot, I narrowed my stability issue to one particular SO-DIMM (as opposed to the slot \u0026ndash; thankfully).\nTomorrow I should be able to send the 1GB back and have it replaced. I\u0026rsquo;ll also try out installing windows (Only if Galactic Civilizations won\u0026rsquo;t work in Windows through Parallels). I\u0026rsquo;ll definitely at least have Ubuntu running on here.\nMore as it comes\u0026hellip;\n","permalink":"/posts/welcome-to-the-21st-century-3/","summary":"\u003cp\u003eSo I purchased a Macbook. I waited until I had it for a bit before posting about it. I\u0026rsquo;ve only had it for two days, but am really enjoying it thus far. I will post photos later.\u003c/p\u003e\n\u003cp\u003eThis is not a review. There are enough of those already.\u003c/p\u003e\n\u003cp\u003eThe screen is excellent, the keyboard only takes a few minutes to get used to, the touch-pad is very natural, and the heat issues are\u0026hellip; intermittent, and very dependent on ambient-temperature. During the day, my machine tends to idle anywhere from 50 - 60 Celsius. Night tends to drop a few degrees to 45 - 52.\u003c/p\u003e","title":"Welcome to the 21st Century"},{"content":"So as part of this little repository I made for myself, I\u0026rsquo;ve built a script to regenerate the the required apt files.\nBut here is the issue: I\u0026rsquo;ve got a symlink called \u0026ldquo;6.06\u0026rdquo; pointing to the \u0026ldquo;dapper\u0026rdquo; directory. I don\u0026rsquo;t wish to regenerate both, as they are identical contents, and ls -d */ returns the symlink as a directory. I\u0026rsquo;ve resorted to using some command-line kung-fu to get around this, but if anybody happens to know of a better way, please let me know:\nls -F | grep \u0026amp;quot;/\u0026amp;quot; | sed -e \u0026amp;quot;s////\u0026amp;quot; Update 2009-09-30 I\u0026rsquo;ve decided to update this a few years after publishing incase anybody comes from google looking for an answer. This was an extremely bad way to do this. What you should be using is find:\nfind . -type d The above runs find in your current directory looking for directories. If you do not want a recursive listing, specify -maxdepth 1.\n","permalink":"/posts/listing-only-directories/","summary":"\u003cp\u003eSo as part of this little repository I made for myself, I\u0026rsquo;ve built a script to regenerate the the required apt files.\u003c/p\u003e\n\u003cp\u003eBut here is the issue: I\u0026rsquo;ve got a symlink called \u0026ldquo;6.06\u0026rdquo; pointing to the \u0026ldquo;dapper\u0026rdquo; directory. I don\u0026rsquo;t wish to regenerate both, as they are identical contents, and \u003ccode\u003els -d */\u003c/code\u003e returns the symlink as a directory. I\u0026rsquo;ve resorted to using some command-line kung-fu to get around this, but if anybody happens to know of a better way, please let me know:\u003c/p\u003e","title":"Listing only Directories"},{"content":"I\u0026rsquo;ve been reading up on debian packages since I\u0026rsquo;m now an Ubuntu user (Ubuntu being derived from Debian). I was previously building packages, copying them around, and manually installing and pinning versions. Today, I decided to simplify this process by creating my own package repository.\nBehold, [http://packages.cidesign.ca/](http://packages.cidesign.ca/ packages.cidesign.ca).\nThe following lines in /etc/apt/sources.list will add this repository:\n## cidesign packages deb http://packages.cidesign.ca/ubuntu/ dapper main deb-src http://packages.cidesign.ca/ubuntu/ dapper main Most of the packages are from using checkinstall, so dependancies are not mapped at all. I\u0026rsquo;ll be going about doing these packages properly at some point in the future (likely when new versions are released).\nAlso note that there are both i386 and powerpc packages here, and that if a package exists for one, it does not neccessarily exist in the other. A good example is Network-Manager, which works fine on my intel laptop with a ipw2100 wireless card, but fails on my ibook with bcm43xx wireless. I built a powerpc version that worked for me, and had no reason to build i386.\nThe howto \u0026ldquo;[Create the Debian Package Repository](http://www.calivia.com/sipxpbx-on-debian-create-the-debian-package-repository Create the Debian Package Repository)\u0026rdquo; is extremely useful, and covers creating a new repository.\n","permalink":"/posts/ubuntu-packaging-goodness/","summary":"\u003cp\u003eI\u0026rsquo;ve been reading up on debian packages since I\u0026rsquo;m now an Ubuntu user (Ubuntu being derived from Debian). I was previously building packages, copying them around, and manually installing and pinning versions. Today, I decided to simplify this process by creating my own package repository.\u003c/p\u003e\n\u003cp\u003eBehold, [http://packages.cidesign.ca/](\u003ca href=\"http://packages.cidesign.ca/\"\u003ehttp://packages.cidesign.ca/\u003c/a\u003e packages.cidesign.ca).\u003c/p\u003e\n\u003cp\u003eThe following lines in /etc/apt/sources.list will add this repository:\u003c/p\u003e\n\u003cpre\u003e\u003ccode\u003e## cidesign packages\ndeb http://packages.cidesign.ca/ubuntu/ dapper main\ndeb-src http://packages.cidesign.ca/ubuntu/ dapper main\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eMost of the packages are from using checkinstall, so dependancies are not mapped at all. I\u0026rsquo;ll be going about doing these packages properly at some point in the future (likely when new versions are released).\u003c/p\u003e","title":"Ubuntu Packaging goodness"},{"content":"I\u0026rsquo;ve replaced the slow Hitachi 4200rpm 2MB cache hard disk in my iBook. Unfortunately the new Samsung 40GB 5400rpm 8MB cache drive seems to have horrible power management features, and gets quite hot. Though it is a very noticable upgrade, system performance is much better now.\nI\u0026rsquo;ve taken this opportunity to perform a fresh install of both OSX and Ubuntu Dapper (using Beta 2 media). I\u0026rsquo;m creating a how-to page detailing special steps needed during or after install.\nObviously I have not written a howto on OSX as it is plainly simple (just remember to turn off the localizations and printer drivers you do not need. It saves you about 2GB\u0026hellip;)\nUpdate 2013-02-26 The how-to page did not survive the transition from Wordpress to Posterous. I noticed this while performing my Posterous exodus to ikiwiki (Damn you Twitter). It\u0026rsquo;s likely lost. Hopefully it will not be missed.\n","permalink":"/posts/linux-on-the-ibook-updates/","summary":"\u003cp\u003eI\u0026rsquo;ve replaced the slow Hitachi 4200rpm 2MB cache hard disk in my iBook. Unfortunately the new Samsung 40GB 5400rpm 8MB cache drive seems to have horrible power management features, and gets quite hot. Though it is a very noticable upgrade, system performance is much better now.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve taken this opportunity to perform a fresh install of both OSX and Ubuntu Dapper (using Beta 2 media). I\u0026rsquo;m creating a how-to page detailing special steps needed during or after install.\u003c/p\u003e","title":"Linux on the iBook updates"},{"content":"I\u0026rsquo;m pretty bad for doing updates.\nI just turned 23 on Tuesday to much fanfare and celebrations (I had dinner at Spageddy Eddy\u0026rsquo;s with Alexis ;)\nMe and Chris K are having a joint birthday celebration at the Elephant \u0026amp; Castle, located on King and Wellington, just inside Galleria Mall. Saturday May 06 2006, about 10:00PM (I\u0026rsquo;ll be there a bit earlier to eat though). Feel free to show up, and bring whomever you would like.\n","permalink":"/posts/birthday-2033/","summary":"\u003cp\u003eI\u0026rsquo;m pretty bad for doing updates.\u003c/p\u003e\n\u003cp\u003eI just turned 23 on Tuesday to much fanfare and celebrations (I had dinner at Spageddy Eddy\u0026rsquo;s with Alexis ;)\u003c/p\u003e\n\u003cp\u003eMe and Chris K are having a joint birthday celebration at the Elephant \u0026amp; Castle, located on King and Wellington, just inside Galleria Mall. Saturday May 06 2006, about 10:00PM (I\u0026rsquo;ll be there a bit earlier to eat though). Feel free to show up, and bring whomever you would like.\u003c/p\u003e","title":"Birthday"},{"content":"Well, I\u0026rsquo;ve gone back to wordpress. No real reason why, however. Typo is some very good software as well.\nI\u0026rsquo;m also running Ubuntu 6.06 \u0026ldquo;Dapper Drake\u0026rdquo; on my server. \u0026ldquo;Oh cool\u0026rdquo; you may say before you realize that it is only 6.04 right now. Let me answer a few questions\n\u0026ldquo;Why Dapper?\u0026rdquo; I wanted to have some neat things like Avahi, a newer mono, and a few other nicities installed. This was looking to be too troublesome with 5.10 \u0026ldquo;breezy\u0026rdquo;.\n\u0026ldquo;Are you crazy? You\u0026rsquo;re running an in-development, unstable operating system on your server?\u0026rdquo; This is really two questions, but I will answer them anyway. First of all, Yes, but I fail to see how this is related to the topic at hand. Secondly, I\u0026rsquo;m not overly concerned with stability on my home server. As long as email works, I\u0026rsquo;m okay.\n","permalink":"/posts/wordpress-and-dapper/","summary":"\u003cp\u003eWell, I\u0026rsquo;ve gone back to wordpress. No real reason why, however. Typo is some very good software as well.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m also running Ubuntu 6.06 \u0026ldquo;Dapper Drake\u0026rdquo; on my server. \u0026ldquo;Oh cool\u0026rdquo; you may say before you realize that it is only 6.04 right now. Let me answer a few questions\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026ldquo;Why Dapper?\u0026rdquo;\u003c/strong\u003e I wanted to have some neat things like Avahi, a newer mono, and a few other nicities installed. This was looking to be too troublesome with 5.10 \u0026ldquo;breezy\u0026rdquo;.\u003c/p\u003e","title":"Wordpress and Dapper"}]