Lighthouse 2.0.000 not displaying content

branko · December 9, 2015, 1:15am

Continuing the discussion from Outernet cubesat news:

Hey Ken. Saw your logs in the inbox, so I thought I’d reply here.

From your log, I see you are in fact receiving data:

Dec  8 21:07:30 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/9.jpg
Dec  8 21:07:34 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/8.jpg
Dec  8 21:07:45 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/2.jpg
Dec  8 21:08:13 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/7.jpg
Dec  8 21:08:24 outernet user.info outernet.monitor: Transmitting buffered data
Dec  8 21:08:24 outernet user.info outernet.monitor: Transmission complete, claring local buffer
Dec  8 21:08:40 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/4.jpg
Dec  8 21:09:20 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/3.jpg
Dec  8 21:09:43 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/1.jpg
Dec  8 21:10:03 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/5.jpg
Dec  8 21:10:15 outernet user.info ondd[1203]: [carousel] completed: World Possible: RACHEL/One Laptop Per Child Collection/Yay-Bee-See/numbers/6.jpg

If you are not seeing any content in the file manager, then it’s some other issue. Are you using your hard drive as external storage? If you’ve used the drive on ORx in the past, you will need to remove a file named .platform on your drive before you can use it with Lighthouse.

If the above does not resolve the issue, please plug in the drive, and then download the system log shortly after the LEDs settle (without unplugging the drive), and post here or to my email (here is preferred so other users can also see if they’re having the same issue, though).

If you can, try to extract /mnt/data/log/fsal.log from the device and post that as well. FSAL is a new component, and we did not have enough time to expose a download button in the settings. We’re working on that atm, but meanwhile it needs to be extracted manually using scp. Let me know if you need assistance with that.

kenbarbi · December 9, 2015, 2:39am

Interesting. My hard drive has been reformatted each time before I connected it to the Lighthouse, and each time my blank hard drive gets these files copied to it:

That said, yes .platform does get recopied to the hard drive, so it must exist on my original Lighthouse (which never operated in an ORx environment).

I deleted .platform and plugged it into the Lighthouse again to see what happened. I rebooted the Lighthouse at 2015-12-09 02:15:19.

The Lighthouse continued to download files, but nothing appears in the Librarian. Also, I stopped the Lighthouse and relooked at the hard drive, and discovered the .platform file got recopied to the hard drive.

I’m sending the app and log files by e mail. Ken

branko · December 9, 2015, 10:22am

Thanks to @bmcintosh for reporting a similar issue, I think I know what the problem is. In some rare cases (I suspect SD card throughput may be a factor), the indexer fails to receive notifications from the downloader. As a temporary remedy, you can manually restart the indexer.

SSH into the device
run sudo /etc/init.d/S91fsal restart
Type in ‘outernet’ as password

Indexing will take a while but you should start seeing at least a couple of folders immediately.

We’re fixing a few other issues as well, so we’ll have a hotfix release soon. This issue will be officially fixed then. If anyone wants to give a hand with testing, I could put development snapshots online.

branko · December 9, 2015, 10:46am

Ken, just in case, and to rule out the possibility of another issue, please SSH into the device and do this:

sudo su
# type in 'outernet' password
cp /mnt/data/log/fsal.log /mnt/data/downloads
sync

Then unplug your hard drive, plug it into your computer, and email me fsal.log which should now be on your hard drive.

kenbarbi · December 9, 2015, 12:17pm

I will do that. But before I plug my hard drive back in, I’ve been operating without the hard drive, and am getting yet another problem

In the tuner setting I get a message saying: An error occurred during the rendering of this plugin.

When I look at the app log I’m seeing the following:

UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xe2 in position 214: ordinal not in range(128)
while idata and ‘\0’ not in idata:
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 73, in read
data = read(sock)
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 105, in send
root = send(payload)
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 226, in get_transfers
files=ipc.get_transfers(),
File “/usr/lib/python2.7/site-packages/librarian_ondd/dashboard_plugin.py”, line 67, in get_context
context.update(self.get_context())
File “/usr/lib/python2.7/site-packages/librarian_dashboard/dashboard.py”, line 88, in render
Traceback (most recent call last):
[2015-12-09 12:10:19 (+0000)] ERROR Plugin rendering failed: ondd
[2015-12-09 12:10:19 (+0000)] DEBUG ONDD: sending payload:
[2015-12-09 12:10:19 (+0000)] DEBUG ONDD: received data: <?xml version="1.0" encoding="UTF-8"?>yes540.88501890668outernet-0
[2015-12-09 12:10:19 (+0000)] DEBUG ONDD: sending payload:
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xe2 in position 207: ordinal not in range(128)
while idata and ‘\0’ not in idata:
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 73, in read
data = read(sock)
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 105, in send
root = send(payload)
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 226, in get_transfers
files=ipc.get_transfers(),
File “/usr/lib/python2.7/site-packages/librarian_ondd/dashboard_plugin.py”, line 67, in get_context
context.update(self.get_context())
File “/usr/lib/python2.7/site-packages/librarian_dashboard/dashboard.py”, line 88, in render
Traceback (most recent call last):
[2015-12-09 12:07:29 (+0000)] ERROR Plugin rendering failed: ondd
[2015-12-09 12:07:29 (+0000)] DEBUG ONDD: sending payload:
[2015-12-09 12:07:29 (+0000)] DEBUG ONDD: received data: <?xml version="1.0" encoding="UTF-8"?>yes540.87501890197outernet-0
[2015-12-09 12:07:29 (+0000)] DEBUG ONDD: sending payload:
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xe2 in position 207: ordinal not in range(128)
while idata and ‘\0’ not in idata:
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 73, in read
data = read(sock)
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 105, in send
root = send(payload)
File “/usr/lib/python2.7/site-packages/librarian_ondd/ipc.py”, line 226, in get_transfers
files=ipc.get_transfers(),
File “/usr/lib/python2.7/site-packages/librarian_ondd/dashboard_plugin.py”, line 67, in get_context
context.update(self.get_context())
File “/usr/lib/python2.7/site-packages/librarian_dashboard/dashboard.py”, line 88, in render
Traceback (most recent call last):
[2015-12-09 12:07:29 (+0000)] ERROR Plugin rendering failed: ondd
[2015-12-09 12:07:29 (+0000)] DEBUG ONDD: sending payload:
[2015-12-09 12:07:29 (+0000)] DEBUG ONDD: received data: <?xml version="1.0" encoding="UTF-8"?>yes540.87501890197outernet-0

Let me know if you want both logs sent by e mail, or this is enough information. Ken

branko · December 9, 2015, 12:39pm

Yeah, that’s a known issue. Sorry about that.

kenbarbi · December 9, 2015, 12:56pm

OK I’ll pass on that issue. I have my hard drive plugged in again and rebooted the Lighthouse. I’m receiving on 4 carousels - 3 of them are skipping, and 1 is downloading Outernet-Rx-2.0.000.pkg

I presume that means I already have downloaded the files being skipped.

I’ll do

sudo su
type in ‘outernet’ password
cp /mnt/data/log/fsal.log /mnt/data/downloads
sync

and send you the file in awhile after the Lighthouse has some time to download files. Thanks, Ken

kenbarbi · December 9, 2015, 1:34pm

When I type this I get the message command not found

username “outernet”.
outernet@192.168.1.16’s password:
outernet@outernet:~$ sudo /etc/init.d/S91fsal
Password:
sudo: /etc/init.d/S91fsal: command not found

I’ll get the fsal.log next.

Ken

branko · December 9, 2015, 1:35pm

Try

sudo /etc/init.d/S90fsal restart

bmcintosh · December 9, 2015, 1:37pm

branko:

Did you login to my Pi and restart indexer? If not, I now see some files coming into the directory. I may have not waited long enough. I left it all running last night and this morning I saw files coming in.
I am certainly willing to do some testing - please feel free to put development snapshots up somewhere, along with a list of fixed items that we can verify. Some simple test instructions or steps would be useful as well. For example, do you want us to reload the SD card from scratch, ext hard drive connected or not, Pi version to test, etc.

You may also want to make it clear that data may be lost/corrupted and/or systems may be bricked until reloaded with the stable release version. (for those who may need their systems to be stable and working)

Cheers,
Brian

branko · December 9, 2015, 1:39pm

I did restart it, yes.

I shall do so then.

Bricking a Pi is very hard to do, as all you need to do is make a new SD card using a known good version. For Lighthouse, it can be fixed with a manufacturing kernel image and SD card. I’ll put that up as well.

branko · December 9, 2015, 1:51pm

Ken, from the logs you’ve sent me, I don’t see any issues with FSAL at all. Can you post some screenshots that demostrates the issue you’re having?

kenbarbi · December 9, 2015, 2:22pm

This is what I see all the time: 3 carousels being skipped, and 1 carousel downloading 2.0.000

When I go to the Library I have no content and see:

I get the same results when I take the hard disk off the system and just run with the basic installed Lighthouse 4 GB SD storage.

Perhaps I need to completely restore the Lighthouse software. To do that, should I download 2.0.000 and follow the manual update procedure? Ken

branko · December 9, 2015, 2:51pm

Do files appear in the file manager?

kenbarbi · December 9, 2015, 2:59pm

I only see the basic files - - no new content ever. Ken

branko · December 9, 2015, 3:00pm

Ok, as far as I see, that’s all old content from the old release. There’s nothing new in there.

branko · December 9, 2015, 3:04pm

Does it work if you disconnect the drive?

EDIT: Sorry, I see you already said it doesn’t.

This is very puzzling.

branko · December 9, 2015, 3:11pm

So, to summarize the events:

plugged in the external drive
content was downloaded but did not show up in the library
you formatted the drive between tries
content stil would not appear, but ONDD says it’s skipping files

Is this correct?

kenbarbi · December 9, 2015, 3:36pm

That’s correct, and yes it happens with or without the hard drive. I looked at the content on the hard drive after each reformat and it’s the “factory default” stuff from 2014. Perhaps I should mention, my Lighthouse worked fine with the old version 1.4.300 before the update to 2.0.000.

Still, can I reload the Lighthouse with an SD chip version of 2.0.000 according to the manual update instructions/ Ken

branko · December 9, 2015, 4:05pm

What I think happened is this:

You started the receiver with the hard drive attached, so all content got written to the new drive. You probably did not wait for the content to be indexed, and you wiped the data off the drive during formatting, but the receiver remembered that you’ve already downloaded the files, so it never downloaded them again. Rinse and repeat a few times until all files have been marked as already completed and now they are all skipping, but you have no physical files on the hard drive or the internal storage.

Otherwise, I don’t see any issues with the software or the hardware (including your hard drive).

I’ll see how you can reset the box to factory state and will let you know.

EDIT: You can reset the downloads like so:

sudo /etc/init.d/S80ondd stop
sudo rm /mnt/data/cache/zindex
sudo /etc/init.d/S80ondd start
sudo /etc/init.d/S90fsal restart

A bit of technical background. Librarian’s file manager doesn’t really touch the storage itself. Instead, there’s a storage indexer called FSAL (filesystem abstraction layer). FSAL performs indexing whenever it is contacted by the downloader abotu completed downloads, then it schedules the indexing, which runs as soon as possible (but not necessarily immediately). When indexing is done, Librarian is notified of new entries, and then Librarian, in turn, indexes content according to metadata. As you can imagine, this takes a while all in all, not like direct physical access to files on your computer. I won’t go into the details of why we chose to implement it this way, but that’s how it works.

Main consequence of this is that:

when content metadata file is not available, you won’t see the content in the library
if content metadata is available, but content files are not there, you will see the item in the library, but you will get a 404 when you try to access the content