notes.txt
author stephen
Thu, 07 Sep 2017 14:02:18 +0100
changeset 63 5c92de4caca6
parent 55 6559c3bacf09
permissions -rw-r--r--
notes on outage
stephen@0
     1
stephen@0
     2
20170111 - device online for 1 min @0948 - someone on roof?
stephen@0
     3
20170112 - device offline at 1524, surprised if that's power? check
stephen@0
     4
20170113 - device work @ 0933 briefly - see early-wake-theory below
stephen@15
     5
           early-wake-theory:
stephen@15
     6
            if device goes below 11.1V while sleeping then it won't wake on
stephen@15
     7
            the RTC but will wake when the solar charger sees 12V again,
stephen@15
     8
            at which point it will see that it ought be asleep unti 1100
stephen@15
     9
            and so it dozes until then - possibly means that we're keeping
stephen@15
    10
            the kerlink up after the eurotech has gone to sleep, otherwise
stephen@15
    11
            why is the voltage still dropping?
stephen@15
    12
            - I modified /etc/stopproc.sh to stop pbm and call ker-off
stephen@15
    13
20170124 - took that out again, added code to pbm binary
stephen@15
    14
           looks like that fix didn't do it, need to test
stephen@15
    15
           some more when at node Monday
stephen@10
    16
20170118 - fixed kerlink not powering off - both s/w and wiring!
stephen@10
    17
           we re-charged unit from mains around 1400-1430, after
stephen@10
    18
           that back to solar, hopefully voltage on 19th will
stephen@10
    19
           behave
stephen@13
    20
20170124 - validated that ker-off does remove voltage from kerlink
stephen@13
    21
           power input; found a code path in pbm that wasn't 
stephen@13
    22
           calling ker-off, testing of that needs to be finished
stephen@13
    23
           when device has power again. Note: left device with
stephen@13
    24
           sleeping only set from 1440 to 1455 so it'll wake 
stephen@13
    25
           early on 25th (not likely to get more power today
stephen@13
    26
           due to sun angle)
stephen@14
    27
20170125 - eek, I'd put the ker-off code in the wrong place;-)
stephen@14
    28
           fixed now I think finally
stephen@15
    29
20170126 - seems to have worked out - node went down @1600
stephen@15
    30
           on 25th with battery @12.52V, awoke on schedule 
stephen@15
    31
           @1102 on 26th with battery @12.49V and weather was
stephen@15
    32
           overcast all morning (that I saw:-) on the 26th,
stephen@15
    33
           node is drawing about 0.9-1.1A around 11am
stephen@15
    34
           probably only safe to fully believe stats from
stephen@15
    35
           here on, though some info can be gleaned from
stephen@15
    36
           earlier data no doubt
stephen@17
    37
20170127 - there was a wake @ 10:02 - might be ok though, node
stephen@17
    38
           went into long standby at 1459 on the 26th so was
stephen@17
    39
           close to out of power (11.34V)
stephen@18
    40
20170128 - looks like basil was rebooted about 17 hours ago 
stephen@18
    41
           (around 1800 on the 27th?) (down is also down)
stephen@18
    42
           so vpn wasn't up - not sure if client will connect
stephen@18
    43
           before it's next reboot, hasn't so far after 5
stephen@18
    44
           mins (vpn manually restarted @ 1140) I'm guessing
stephen@18
    45
           there was a power off event
stephen@18
    46
20170201 - basil reboot: there was a bridge0 that was ifup'd
stephen@18
    47
           and that might be interfering somehow, took that
stephen@18
    48
           down, we'll see @ 1100 if that helps, if not, I'll
stephen@18
    49
           wanna go check unit physically (also could be 
stephen@18
    50
           SIM card crap, check that too - SIM seems ok, has
stephen@18
    51
           7GB and 8 days left) 
stephen@18
    52
           Turns out that the SIM was not ok - the "add on"
stephen@18
    53
           thing is not being used and the balance was down
stephen@18
    54
           to 0.65c or actually maybe negative. Topped the
stephen@18
    55
           fecker up.
stephen@18
    56
           Clock was still on jan 29 today, odd - maybe
stephen@18
    57
           rtc isn't working right? also that might interact
stephen@18
    58
           with vpn, not sure - actually no, it was that
stephen@18
    59
           the node thought it was 1600 on the 29th so it
stephen@18
    60
           went to sleep, wonder when it'd have awoken if
stephen@18
    61
           Kerry hadn't hit the physical switch?
stephen@18
    62
           I have the kerlink boottimes.txt that could be
stephen@18
    63
           used to fix the boot times and hence battery
stephen@18
    64
           log of the loradtn node if we want
stephen@19
    65
20170202 - odd timing - node woke @1047 - is power still being
stephen@19
    66
           consumed?
stephen@19
    67
           also odd vpn pattern - new client IPs every few
stephen@19
    68
           minutes, check node's syslog later
stephen@19
    69
20170206 - looks nominal at the moment, nice weekend weather
stephen@19
    70
           helps:-)
stephen@23
    71
20170213 - VPN went down at 1427 in good bright weather with
stephen@23
    72
           battery nicely high  (13.28V at 1357), GSM balance
stephen@23
    73
           also fine, but maybe it's the add-on fecking thing
stephen@23
    74
           I added 3GB of data (good 'till March 15) which'll
stephen@23
    75
           take effect at next connect (reboot?) it didn't
stephen@23
    76
           reconnect in 5 mins anyway. Or, it could just be
stephen@23
    77
           a DHCP lease meets VPN issue. Check tomorrow.
stephen@24
    78
20170214 - yep, yesterday's outage was gsm related - pppd saw
stephen@24
    79
           a disconnection. Device stayed powered up 'till
stephen@24
    80
           1600 as planned, but had no n/w. Once it powered
stephen@24
    81
           up today, it was back on line. Must check that
stephen@24
    82
           again on March 10th (set a reminder:-)
stephen@24
    83
           Meanwhile no sun today, so node will likely go
stephen@24
    84
           to sleep in an hour or so (now @12.27V and drawing
stephen@24
    85
           0.9A) 
stephen@29
    86
20170222 - seems like there is some browning out happening,
stephen@29
    87
           today and on the 18th - might be the higher load
stephen@29
    88
           means that the thresholds aren't right for this
stephen@29
    89
           setup with voltage is marginal (e.g. 11.5V)
stephen@32
    90
20170228 - graphs show node up yesterday but I didn't see 
stephen@32
    91
           that from CLI, check it out
stephen@32
    92
           also - redo scripts to add March
stephen@38
    93
20170308 - we added the code for the 2nd ammeter for the 
stephen@38
    94
           device under test (DUT). Noted that the voltage
stephen@38
    95
           calibration may need a bit of work, but not
stephen@38
    96
           enough to make a change now as that'd invalidate
stephen@38
    97
           data since Jan. Worth doing more before any 
stephen@38
    98
           further deployment. The new pbm code was tested
stephen@38
    99
           on the bench and deployed to the rooftop node
stephen@38
   100
           at 1500 today. We'll follow up with the new
stephen@38
   101
           phidgets/veroboard assembly shortly (so DUT
stephen@38
   102
           figures for now will be garbage) some voltages
stephen@38
   103
           seen on the bench power supply and as seen by
stephen@38
   104
           pbmd are:
stephen@38
   105
              Bench                 pbmd
stephen@38
   106
              11.6                  11.87
stephen@38
   107
              11.8                  12.10
stephen@38
   108
              12.8                  12.97
stephen@38
   109
              13.2                  13.44
stephen@38
   110
           Current measurements OTOH, seem accurate 
stephen@38
   111
           enough to within 100mA or so.
stephen@43
   112
20170310 - veroboard with new ammeter deployed today (so 
stephen@39
   113
           some outage this morning) DUT readings showing
stephen@39
   114
           about 300+/-40mA
stephen@43
   115
20170317 - added a graphic showing the DUT power consumption
stephen@43
   116
           vs. the overall power log
stephen@45
   117
20170321 - extended sripts to cover to end-April
stephen@45
   118
           also noticed that SPIKE readings are being generated
stephen@45
   119
           when voltage is identical in two successive readings
stephen@45
   120
           probably a pbm bug, but not a biggie
stephen@48
   121
20170408 - topped up SIM card - balance was small and device
stephen@48
   122
           offline on a sunny day, see if fixed with tomorrow's
stephen@48
   123
           reboot
stephen@48
   124
20170409 - seems like that's it, up nomimally this am, and
stephen@48
   125
           past three days readings seem corrupted (no NTP to
stephen@48
   126
           set date, odd that it falls back to the last "good"
stephen@48
   127
           day, something to check out)
stephen@53
   128
20170416 - seems like the root FS is full on DTN node, note
stephen@53
   129
           sure since when, cleaning today. Also can't ping
stephen@53
   130
           the kerlink, not sure if connected. Need a way 
stephen@53
   131
           for unattended upgrades to not fill / based on
stephen@53
   132
           adding stuff to /boot. Fixed that manually and
stephen@53
   133
           did a reboot at 1500 IST... unit came back up
stephen@53
   134
           fine, but still no sign of kerlink. Will have to
stephen@53
   135
           go over and see Tuesday
stephen@53
   136
           oops - node went offline in the middle of an
stephen@53
   137
           apt get upgrade about 1530 local, not sure if 
stephen@53
   138
           power (should be enough, I'd have thought) maybe
stephen@53
   139
           vpn, check tomorrow
stephen@53
   140
20170417 - further checking... unit came up @1100 nominal
stephen@53
   141
           last time kerlink asked for dhcp was march 25
stephen@53
   142
           finish update for dtn node then check dhcp is
stephen@53
   143
           running (looks like it might not be)
stephen@53
   144
           Ah-ha! kerlink has moved to .17 from .11 for
stephen@53
   145
           some reason, and syslog I copied down yesterday
stephen@53
   146
           seems odd, more when endlless update done
stephen@53
   147
20170418 - nominal, still not sure what's up with syslog
stephen@53
   148
           but will look tomorrow
stephen@55
   149
20170516 - refreshed vodafone a/c, was offline for a week
stephen@55
   150
           or two before that, back now
stephen@63
   151
20170831 - device down for a couple of weeks (since TBD),
stephen@63
   152
           went over, possible short on power input, rewired
stephen@63
   153
           that and battery seems to be charging (at ~10.5V now)
stephen@63
   154
		   will check and fix if needed
stephen@63
   155
20170906 - replaced a bit more wiring, still not clearly
stephen@63
   156
           charging, will check again and bring to lab
stephen@63
   157
           if not
stephen@38
   158
stephen@48
   159
stephen@53
   160