Answers.Body: <p>I think you should be able to do this. The screen is represented by a <a href="https://www.kernel.org/doc/Documentation/fb/framebuffer.txt" rel="nofollow noreferrer">framebuffer</a> device. In Linux, <em>everything is a file</em>, so even the framebuffer is a file, namely <code>/dev/fb0</code>.</p>
  
  <p>So, you should be able to read the display by reading the <code>/dev/fb0</code> file using something like this:</p>
  
  <p>EDIT: Using direct commands didn't work out so well, so here is a sample using system commands.</p>
  
  <pre><code>Getting file content
  --------------------
  
  Used to upload datalog files - file handle is only closed when reaching EOF
  and file is not open for writing
  
  BEGIN_GETFILE:
  
    Bytes send to the brick:
  
    xxxxxxxx0196xxxxxxx
    bbbbmmmmttssllllnnn...
  
    bbbb = Bytes in massage, mmmm = message counter, tt = type of command,
    ss = system command, llll = max bytes to read, nnnn.... = path
  
  
    Bytes send to the PC:
  
    xxxxxxxx039600xxxxxxxx00xxx
    bbbbmmmmttssrrllllllllhhppp...
  
    bbbb = bytes ion massage, mmmm = message counter, tt = type of command,
    ss = system command, rr = return status, llllllll = File size, hh = Handle,
    ppp... = payload
  
  
  CONTINUE_GETFILE:
  
    Bytes send to the brick:
  
    0700xxxx019700xxxx
    bbbbmmmmttsshhllll
  
    bbbb = bytes in massage, mmmm = message counter, tt = type of command,
    ss = system command, hh = handle, llll = max bytes to read
  
  
    Bytes send to the PC:
  
    xxxxxxxx039700xxxxxxxx00xxx
    bbbbmmmmttssrrllllllllhhppp...
  
    bbbb = bytes in message, mmmm = message counter, tt = type of command,
    ss = system command, rr = return status, llllllll = File size,
    hh = Handle, ppp... = payload
  </code></pre>
  
  <p><s>Opcodes come from <a href="http://ev3.fantastic.computer/doxygen/cMemory.html" rel="nofollow noreferrer">here</a></s> System commands come from <a href="https://bricks.stackexchange.com/questions/7140/direct-system-command-for-reading-the-display-buffer-of-ev3-bricks/7143?noredirect=1#comment8796_7143">here</a>. The size returned may be 0 since this is not a real file, but the framebuffer size is 7680 bytes (comes from <a href="https://github.com/mindboards/ev3sources/blob/master/extra/linux-03.20.00.13/drivers/video/st7586fb.c#L96" rel="nofollow noreferrer">here</a>, i.e. <code>(178+2)/3*128</code>). Also, the maximum number of bytes allowable for <code>max bytes to read</code> depends on the connection type (USB, Bluetooth or Wi-Fi). 1000 should be a safe number.</p>
  
  <p>Now, hopefully you have the data, but you still need to decode it. The framebuffer has an unusual format where there are 3 pixels per byte (see <a href="http://pdf1.alldatasheet.com/datasheet-pdf/view/326182/SITRONIX/ST7586S.html" rel="nofollow noreferrer">datasheet</a>).</p>
  
  <p>Assuming you have packed all of the data read into a single 7680 byte array you can use nested for loops to convert the data to whatever image format you like. For simplicity, I am going to use a <a href="https://en.wikipedia.org/wiki/Netpbm_format" rel="nofollow noreferrer">.pbm</a>. Just doing this in pseudo code...</p>
  
  <pre><code>// framebuffer data - 60 bytes/row * 128 rows
  buffer[60*128]
  
  // write the PBM header
  write_line("P1")
  write_line("178 128")
  for (row = 0; row &lt; 128; row++) {
      // last byte in each row only has 1 pixel, so only loop first 59 bytes
      for (offset = 0; offset &lt; 59; offset++) {
          var b = buffer[row * 60 + offset];
          // 3 pixels per byte
  
          // bit 7 (or 6) is pixel 0
          write((b &amp; 128) ? "1 " : "0 ")
          // bit 4 (or 3) is pixel 1
          write((b &amp; 16) ? "1 " : "0 ")
          // bit 1 (or 0) is pixel 2
          write((b &amp; 2) ? "1 " : "0 ")
      }
      // the last pixel
      b = buffer[row * 60 + offset]
      write_line((b &amp; 128) ? "1" : "0")
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-25T06:41:54.983Z
Answers.Text: I tried getting OPEN_READ/READ_BYTES working with an actual file first. OPEN_READ returns the correct file size however the handle is always 1 (0x01 0x0 0x0 0x0 in little endian) no matter how many times I make the call. If I subsequently make a call to GET_HANDLE to get the handle for the same file path, I get (0xFF 0xFF 0x0 0x0) for the file handle (instead of 1). Both commands succeed. When I try to actually read the file with READ_BYTES (trying both handles: 0x1 and 0xFFFF), it fails. Something is not right.
Answers.UserDisplayName: Matt
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-25T18:40:15.520Z
Answers.Text: Look's like I made at least one mistake, Handle is 16-bit. Here is a sample that may help: https://github.com/mindboards/ev3sources/blob/78ebaf5b6f8fe31cc17aa5dce0f8e4916a4fc072/lms2012/lmssrc/tst/tst_save.lms#L628
Answers.UserDisplayName: David Lechner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-25T18:53:09.163Z
Answers.Text: Looking at the source code, it looks like if a file fails to open, the command will still return successfully, although I would expect the handle to be 0 and not 1 as you observed. Source: https://github.com/mindboards/ev3sources/blob/78ebaf5b6f8fe31cc17aa5dce0f8e4916a4fc072/lms2012/c_memory/source/c_memory.c#L1251
Answers.UserDisplayName: David Lechner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-25T23:03:44.367Z
Answers.Text: I was able to get things to work by sending the OPEN_READ/READ_BYTES commands at once with a "compound command" (as the docs call it on page 26 of the EV3 Communication Developer Kit PDF), rather than individually. This even works with /dev/fb0, however I cannot read more than 1024 bytes now since the global variable space is 10-bit. If there was an opcode for seeking to a different position (there is not), we could go through the OPEN_READ/READ_BYTES cycle again, this time starting at a different offset.
Answers.UserDisplayName: Matt
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-26T02:15:24.107Z
Answers.Text: Could you save the file contents to an array variable and then read the the array 1000 or 1024 bytes at a time using `opARRAY(READ_CONTENT, ...)`?
Answers.UserDisplayName: David Lechner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-26T02:21:04.483Z
Answers.Text: Actually, perhaps even better, what about the `BEGIN_DOWNLOAD` and `CONTINUE_DOWNLOAD` system commands?
Answers.UserDisplayName: David Lechner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-26T02:49:36.473Z
Answers.Text: Oops. Download is the wrong direction. I update my answer to use the `BEGIN_GETFILE` and `CONTINUE_GETFILE` system commands.
Answers.UserDisplayName: David Lechner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-12-24T16:55:12.460Z
Answers.LastActivityDate: 2015-12-26T02:48:58.117Z
Answers.LastEditDate: 2017-04-13T12:47:56.390Z
Answers.UserDisplayName: David Lechner
Body: <p>I recall being able to read in the display buffer of NXT 2.0 bricks over Bluetooth/USB by issuing Read IO Map system commands. I've looked through the bytecodes and communication documentation for the EV3, but it's not clear how to do this with the EV3. Does anyone have any pointers?</p>
  
Tags: <mindstorms><ev3><programming>
Title: Direct/system command for reading the display buffer of EV3 bricks?
UserDisplayName: Matt
ThreadText: # Direct/system command for reading the display buffer of EV3 bricks?
  
  (Asked by: username_0 on 2015-12-24)
  
  I recall being able to read in the display buffer of NXT 2.0 bricks over Bluetooth/USB by issuing Read IO Map system commands. I've looked through the bytecodes and communication documentation for the EV3, but it's not clear how to do this with the EV3. Does anyone have any pointers?
  
  **Tags:** mindstorms, ev3, programming
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2015-12-24. Score: 3
  
  I think you should be able to do this. The screen is represented by a framebuffer device. In Linux, everything is a file, so even the framebuffer is a file, namely /dev/fb0.
  So, you should be able to read the display by reading the /dev/fb0 file using something like this:
  EDIT: Using direct commands didn't work out so well, so here is a sample using system commands.
  Getting file content
  --------------------
  
  Used to upload datalog files - file handle is only closed when reaching EOF
  and file is not open for writing
  
  BEGIN_GETFILE:
  
    Bytes send to the brick:
  
    xxxxxxxx0196xxxxxxx
    bbbbmmmmttssllllnnn...
  
    bbbb = Bytes in massage, mmmm = message counter, tt = type of command,
    ss = system command, llll = max bytes to read, nnnn.... = path
  
  
    Bytes send to the PC:
  
    xxxxxxxx039600xxxxxxxx00xxx
    bbbbmmmmttssrrllllllllhhppp...
  
    bbbb = bytes ion massage, mmmm = message counter, tt = type of command,
    ss = system command, rr = return status, llllllll = File size, hh = Handle,
    ppp... = payload
  
  
  CONTINUE_GETFILE:
  
    Bytes send to the brick:
  
    0700xxxx019700xxxx
    bbbbmmmmttsshhllll
  
    bbbb = bytes in massage, mmmm = message counter, tt = type of command,
    ss = system command, hh = handle, llll = max bytes to read
  
  
    Bytes send to the PC:
  
    xxxxxxxx039700xxxxxxxx00xxx
    bbbbmmmmttssrrllllllllhhppp...
  
    bbbb = bytes in message, mmmm = message counter, tt = type of command,
    ss = system command, rr = return status, llllllll = File size,
    hh = Handle, ppp... = payload
  
  Opcodes come from here System commands come from here. The size returned may be 0 since this is not a real file, but the framebuffer size is 7680 bytes (comes from here, i.e. (178+2)/3*128). Also, the maximum number of bytes allowable for max bytes to read depends on the connection type (USB, Bluetooth or Wi-Fi). 1000 should be a safe number.
  Now, hopefully you have the data, but you still need to decode it. The framebuffer has an unusual format where there are 3 pixels per byte (see datasheet).
  Assuming you have packed all of the data read into a single 7680 byte array you can use nested for loops to convert the data to whatever image format you like. For simplicity, I am going to use a .pbm. Just doing this in pseudo code...
  // framebuffer data - 60 bytes/row * 128 rows
  buffer[60*128]
  
  // write the PBM header
  write_line("P1")
  write_line("178 128")
  for (row = 0; row < 128; row++) {
      // last byte in each row only has 1 pixel, so only loop first 59 bytes
      for (offset = 0; offset < 59; offset++) {
          var b = buffer[row * 60 + offset];
          // 3 pixels per byte
  
          // bit 7 (or 6) is pixel 0
          write((b & 128) ? "1 " : "0 ")
          // bit 4 (or 3) is pixel 1
          write((b & 16) ? "1 " : "0 ")
          // bit 1 (or 0) is pixel 2
          write((b & 2) ? "1 " : "0 ")
      }
      // the last pixel
      b = buffer[row * 60 + offset]
      write_line((b & 128) ? "1" : "0")
  }
  
  > **Comments:**
  > **username_0**: I tried getting OPEN_READ/READ_BYTES working with an actual file first. OPEN_READ returns the correct file size however the handle is always 1 (0x01 0x0 0x0 0x0 in little endian) no matter how many times I make the call. If I subsequently make a call to GET_HANDLE to get the handle for the same file path, I get (0xFF 0xFF 0x0 0x0) for the file handle (instead of 1). Both commands succeed. When I try to actually read the file with READ_BYTES (trying both handles: 0x1 and 0xFFFF), it fails. Something is not right.  
  > **username_1**: Look's like I made at least one mistake, Handle is 16-bit. Here is a sample that may help: https://github.com/mindboards/ev3sources/blob/78ebaf5b6f8fe31cc17aa5dce0f8e4916a4fc072/lms2012/lmssrc/tst/tst_save.lms#L628  
  > **username_1**: Looking at the source code, it looks like if a file fails to open, the command will still return successfully, although I would expect the handle to be 0 and not 1 as you observed. Source: https://github.com/mindboards/ev3sources/blob/78ebaf5b6f8fe31cc17aa5dce0f8e4916a4fc072/lms2012/c_memory/source/c_memory.c#L1251  
  > **username_0**: I was able to get things to work by sending the OPEN_READ/READ_BYTES commands at once with a "compound command" (as the docs call it on page 26 of the EV3 Communication Developer Kit PDF), rather than individually. This even works with /dev/fb0, however I cannot read more than 1024 bytes now since the global variable space is 10-bit. If there was an opcode for seeking to a different position (there is not), we could go through the OPEN_READ/READ_BYTES cycle again, this time starting at a different offset.  
  > **username_1**: Could you save the file contents to an array variable and then read the the array 1000 or 1024 bytes at a time using `opARRAY(READ_CONTENT, ...)`?  
  > **username_1**: Actually, perhaps even better, what about the `BEGIN_DOWNLOAD` and `CONTINUE_DOWNLOAD` system commands?  
  > **username_1**: Oops. Download is the wrong direction. I update my answer to use the `BEGIN_GETFILE` and `CONTINUE_GETFILE` system commands.  
  
  ---
  
  