MityCAMAC

Links

• MityARM
• https://bitbucket.org/ttriumfdaq/gpmc-camac/src/master/
• https://bitbucket.org/ttriumfdaq/gpmc-camac-aux-firmware/src/master/
• inventory https://ladd00.triumf.ca/~daqinv/inventory3/public/index.php/daqinv/lists/162
• (dead) redmine project https://edev.triumf.ca/projects/edevel00195
• (dead) svn checkout https://edev.triumf.ca/svn/edevel00195
• (obsolete) firmware sources: see mess in daq00:/home/olchansk/daq/CAMAC-Crate-Controller. hint: never converted from svn to git.

Getting Started

• For general information on the Critical Link ARM SOM, start here MityARM
• Follow instruction at MityARM to login into Linux
• Build and install GPMC-CAMAC drivers
• Load the FPGA firmware
• Run CAMAC tests

BOOTCONFIG

BOOTCONFIG settings:

• CL eval board:
  • BOOTCONFIG[0..11] are SYSBOOT[0..11]
  • boot from SD: all open, [3] closed
  • boot from on-SoM NAND: all open, close [2], [3], [8], [9], [10], [11] (6 jumpers)

• CAMAC board Rev1:
  • top-to-bottom: B0..4,G0..5,R0 correspond to LCD_DATA[0..11] are SYSBOOT[0..11]
  • boot from SD: all open, B3 closed
  • boot from on-SoM NAND: all open, close B2, B3, G3, G4, G5, R0 (6 jumpers)

NOTES:

• boot from EMAC1 requires PHY connection to RGMII1, CLeval and CAMAC boards connected PHY to RGMII2.

SYSBOOT modes: http://ladd00.triumf.ca/~olchansk/MityARM/Docs/spruh73g.pdf table 26-7

SYSBOOT[4..0]
00000 - reserved
00001 - UART0
00010 - UART0
00011 - UART0
00100 - UART0
00101 - UART0
00110 - EMAC1, SYSBOOT[7..6]: 00=MII, 01=RMII, 10=reserved, 11=RGMII
00111 - EMAC1
01000 - EMAC1
01001 - EMAC1
01010 - EMAC1
01011 - USB0
01100 - USB0
01101 - USB0
01110 - reserved
01111 - fast external boot
10000 - XIP
10001 - XIP
10010 - NAND
10011 - NAND
10100 - NAND
10101 - NAND-I2C
10110 - SPI0
10111 - MMC0 <---- default (boot from SD flash) - other bits - don't care
11000 - SPI0
11001 - SPI0
11010 - XIP
11011 - XIP
11100 - MMC1
11101 - reserved
11110 - reserved
11111 - fast external boot

GPMC interface

0x8 = CS0 - on-SoM NAND flash (boot loader)
0x9 = CS1 - not used
0xa = CS2 - not used
0xb = CS3 - CAMAC interface

./gpmc 0x0a000000 0x12ff5678

CAMAC interface

General information:

• CAMAC interface responds to GPMC CS3
• the host CPU should always issue 32-bit read and write cycles on even addresses (last 2 bits of address set to 0)
• address bits 15..12 address the FPGA GPMC registers (4 bits, 16 registers)
• remaining address bits are used for encoding CAMAC operations
• all GPMC registers are 32-bit wide

LEDs (Rev1)

                       ----
gpmc            0x0008 |OO| 0x0040 ACB RQ (lemo output "request")
busy            0x0010 |OO| 0x0080 grant_in (lemo input)
ACB RQ          0x0020 |OO| 0x0100 ACB RI
-----------------------------------------------------

Front Panel LEMO connectors (Rev1)

top
|
| busy out
| inhibit in
| grant in
| grant out
| request out
|
bottom

Install the software

• Get and build the software tools

ssh daquser@armdaq02
mkdir packages
cd packages
git clone https://bitbucket.org/ttriumfdaq/gpmc-camac.git
cd gpmc-camac
make CROSS_COMPILE=
ls -l gpmc camac srunner_gpmc

-bash-4.2$ ls -l gpmc camac srunner_gpmc
-rwxrwxr-x 1 daquser users 668337 Dec  6 22:43 camac
-rwxrwxr-x 1 daquser users 654454 Dec  6 22:43 gpmc
-rwxrwxr-x 1 daquser users 692961 Dec  6 22:43 srunner_gpmc
-bash-4.2$ 

• build kernel driver

cd gpmc-camac/kernel-module
make CROSS_COMPILE=
ls -l gpmc_camac_module.ko

-bash-4.2$ ls -l gpmc_camac_module.ko
-rw-rw-r-- 1 daquser users 93154 Dec  6 22:48 gpmc_camac_module.ko
-bash-4.2$ 

• if /daq/daqshare/olchansk is not available, building the kernel module will fail. To proceed, get the correct kernel sources and cross-compiler:

cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/linux-mityarm-335x.git
git clone https://dd1@bitbucket.org/ttriumfdaq/u-boot-mityarm-335x.git # not needed for building kernel module
cd gpmc-camac/kernel-module
edit Makefile, change KDIR to $HOME/packages/linux-mityarm-335x and CROSS_COMPILE=location of cross-compiler tools

• Note: gpmc-camac kernel module for Linux kernel uImage-KO10 (uname -a "Linux armdaq02.triumf.ca 3.2.0-00377-gb23ecfb-dirty #33 Sun Nov 13 05:10:32 UTC 2016 armv7l armv7l armv7l GNU/Linux") *MUST* be built using native gcc (make CROSS_COMPILE=).

• install kernel drivers

cd $HOME/packages/gpmc-camac/kernel-module
make install
su - ### login as root
### follow instructions printed by "make install"

• verify that debugfs is mounted on /sys/kernel/debug: ls -l /sys/kernel/debug/omap_mux/

• run /etc/rc.local or reboot

Firmware revisions

Run "./camac" to read the firmware revision number (reg0)

• 0x14120723 - ???
• 0x15012159 - working Rev1, old/new "done" bit.
• 0x15120106 - Rev2 AUX (ok on Rev1 board)
• 0x15120946 - Rev2 MASTER
• 0x25022815 - Rev2 AUX with working arbitration
• note: LAM register does not seem to work in these firmware (register 7 LAM bits are always 1)

Firmware update

• test the active serial flash programmer interface

$ ./srunner_gpmc -id -16 /dev/null

Operation: Identify EPCS Device
bb_open!
Info: Found GPMC 0xABCD FlashProgrammer interface.

Info: Silicon ID - 0x14 
Info: Serial Configuration Device - EPCS16

Operation Completed!!!

• write pof file to flash memory

$ ./srunner_gpmc -program -16 CAMAC_Crat_Controller.pof

Operation: Programming EPCS
Info: Programming file: "CAMAC_Crat_Controller.pof" opened.
Info: File size: 2097359 bytes.
Skipping POF file header: start of data at 154, file size 2097359, remain 2097205, modulo 1024 = 53
Final file size 2097152
bb_open!
Info: Found GPMC 0xABCD FlashProgrammer interface.

Info: Silicon ID - 0x14 
Info: Serial Configuration Device - EPCS16
Info: Erasing...
....................Info: Erase Done
Info: Start programming process.

Info: Programming 8192 pages...
Info: Programming successful
bb_close!

Operation Completed!!!

• reboot the FPGA

lxdaq30:LINUX$ ./camac --reboot
CAMAC firmware: 0x14120407
rebooting FPGA...
CAMAC firmware: 0x14120711
lxdaq30:LINUX$ 

Register 1 - command

Write-only 32 bit register:

command 1: CMD_reset: 0x
command 2: CMD_reconfigure
command 3: CMD_TestMode
command 4: CMD_Zero
command 5: CMD_Clear
command 6: CMD_Inhibit
command 7: CMD_WriteLED: 0xXXXX0007 -> write 0xXXXX to the LED bus

Register 2 - test read/write register

32-bit read/write register for testing GPMC bus communications

Register 3 - FPGA flash programmer

active-serial flash programmer interface (see srunner, srunner_vme and srunner_gpmc).

Register 4 and 5 - CAMAC command, write data and read data

A write cycle to the GPMC bus carries 16 bits of address and 32 bits of data:

Address 16 bits (Quartus cmd_A[15..0])
----
15..12 - GPMC register address 
--
11 - not used
10 - not used
9 - not used
8..4 - CAMAC N[4..0]
--
3 - CAMAC F[4]
2 - CAMAC_start - set this bit to 1 to trigger a CAMAC operation. Value 0 is for testing the communications without running the CAMAC sequencer.
1 - not accessible
0 - not accessible
----

Data 32 bits: (Quartus cmd_D[31..0])
----
31..28 - CAMAC F[3..0]. (CAMAC F[4] is cmd_A[3])
27..24 - CAMAC A[3..0]
--
23..0 - CAMAC write lines W[24..1]
----

To verify correct data transmission on the GPMC bus and correct latching of the data in the CAMAC part of the FPGA, the 32 bits of data can be read back through register 4 and the 16 bits of address can be read through the low 16 bits of register 5. Reading from these registers is not necessary for operating CAMAC cycles.

Register 6 - CAMAC status and read data

Read-only 32 bit register:

31 - camac_done (from ???) - CAMAC state machine has finished
30 - arbitration_timeout (from 0x25022050) - CAMAC arbitration timeout (RQ line stuck at 1, GRANT-IN not connected)
29 - idle_timeout (from 0x25022200) - at the end of CAMAC cycle, timeout waiting for bus to be in idle state (RQ stuck at 1, GRANT-IN stuck at 1).
28 - BUSY_OUT
--
27 - BUSY
26 - set to 1 when CAMAC cycle is finished and new data is ready
25 - CMD_ACCPT
24 - RESPONSE
--
23..0 - CAMAC read data[24..1]

Register 7 - CAMAC LAM and interface status

Read-only 32-bit register:

31 - EXT_BUSY
30 - EXT_INHIBIT
29 - EXT_GRANT_IN
28 - EXT_GRANT_OUT
--
27 - ACB ACL
26 - ACB RI
25 - ACB RQ
24 - not used
--
23..0 - CAMAC LAM from slots 24..1

Software commands

./gpmc # initialize the GPMC interface
./gpmc 0x0b000000 # read firmware revision
./gpmc 0x0b001000 2 # reconfigure the fpga
./srunner_gpmc -id -64 /dev/null # use the flash programmer to read the flash chip identity
./srunner_gpmc -program -16 ../QUARTUS/output_files/CAMAC_Crat_Controller.pof # write POF file into the flash (remember to do an FPGA reconfigure to reboot into the new firmware)
./gpmc 0x0b001000 0x01f80007 # write to the LEDs
./gpmc 0x0b0040ac 0x00123456 # run some camac cycle
./gpmc 0x0b001000 0x80000003 # enable test mode
./gpmc 0x0b004174 0x00123456 # read from slot 23 - "manual input register"
./gpmc 0x0b00416c 0x00000000 # write to slot 22 - dataway display
./gpmc 0x0b00416c 0x00ffffff # write to slot 22 - dataway display
./gpmc 0x0b006000 # read CAMAC read data and status
./gpmc 0x0a00012c # reconfigure FPGA thorugh the CS2 line

./gpmc # initialize the GPMC interface
./camac # read firmware revision, etc
./camac --reboot # reboot the FPGA
./camac --scan # scan CAMAC crate
./camac --gpmctest1 # test GPMC bus integrity - read/write test
./camac --gpmctest2 # test GPMC bus integrity - both data and address read/write test
./camac --testleds # test front panel LEDs - they should blink at 1Hz
./camac --testlemo # test front panel LEMOs - the voltages should alternate between 0V and ???V at 1Hz
./camac r # read CAMAC register r, e.g. \"./camac 0\" to read register 0 - firmware revision
./camac w v # write CAMAC register w with value v, e.g. \"./camac 2 0xabcd\" to write 0xabcd to test register 2
./camac n f a w # run a CAMAC cycle with given N, F, A, write data W

List of on-board hardware

• on SoM: http://www.mitydsp.com/products-services/cpu-engines/mityarm-3359 http://ladd00.triumf.ca/~olchansk/MityARM/Docs/MityARM-335x_Datasheet.pdf
  • TI 3354/3359 CPU: http://ladd00.triumf.ca/~olchansk/MityARM/Docs/spruh73g.pdf
  • 256MB or 512MB NAND flash: GPMC CS#0, 8-bit wide
  • 8MB serial NOR flash: SPI1 interface
  • configuration EPROM: I2C1 interface (module serial number, ethernet MAC, etc)
  • 256MB DDR2 or 512MB DDR3 RAM, 16-bit interface, 300 or 400MHz
  • power management chip tps65910: I2C1, I2C2 interface, http://ladd00.triumf.ca/~olchansk/MityARM/Docs/tps65910.pdf
• on CAMAC board:
  • Micrel KSZ9021RL GigE ethernet PHY: Davinci_MDIO interface http://ladd00.triumf.ca/~olchansk/MityARM/Docs/ksz9021rl-rn_ds.pdf
  • Altera Cyclone4 FPGA EP4CE30F23: GPMC bus

CAMAC ACB arbitration

Note on arbitration between multiple CAMAC controllers using the auxiliary controller bus and the request->grant-in->grant-out method.

• description in the CAMAC IEEE standard is correct, it works and it has no deadlock or other defect
• most important is that action is taken on the GRANT-IN transition 0->1, not on GRANT-IN level
• general sequence, 1st controller grabs the bus:

- camac state machine starts
- wait for RI and GRANT-IN idle (0) (RQ is also usually 0, but other controller may drive it)
- raise RQ
- wait for GRANT-IN 0->1
- drive GRANT-OUT 0 (deny 2nd controller bus access: 2nd controller will never see a GRANT-IN 0-1 transition)
- drive RI (we own the bus)
- drop RQ (GRANT-IN will also drop)
- perform CAMAC operations
- wait for RQ and GRANT-IN idle (other controller should have dropped them, we dropped them, if pull-up resistor is missing, RQ can take ~6 us to float to 0). use debounced RQ and GRANT-IN.
- drop RI
- bus is now in idle state, RI, RQ, GRANT-IN are 0.

• 2nd controller, lost arbitration

- camac state machine starts
- wait for RI (and GRANT-IN), RI=1 means 1st controller has the bus
- RI=0 means 1st controller finished, released the bus
- raise RQ
- wait for GRANT-IN 0->1
- if RI goes up, 1st controller grabbed the bus
- drop RQ
- go back to wait for RI

• why collisions are impossible:

- 1st controller first, 2nd controller later: 2nd controller will see RI, will back off
- 2nd controller first, 1st controller later: 1st controller will see RI, will back off
- both controllers raise RQ at the same time: RQ is plugged into 1st controller GRANT-IN, only it will see GRANT-IN 0->1 transition and grab the bus. 1st controller will drive GRANT-OUT to 0, 2nd controller will never see GRANT-IN 0->1, will see RI go up and will back off.

• in other words, why waiting for GRANT-IN 0->1 cannot deadlock:

RQ is connected to GRANT-IN of 1st controller (and through it's GRANT-IN->GRANT-OUT) to GRANT-IN of 2nd controller
we do not know if we are 1st or 2nd controller
we raise RQ and wait for GRANT-IN 0->1 transition
if we do not see it:
- 1st controller: because RQ->GRANT-IN jumper is missing
- 2nd controller: because 1st controller grabbed the bus and is driving GRANT-OUT 0. we will see them drive RI up and we will back off and try again
- 3rd controller: same as 2nd controller
if we see it:
- 2nd controller: because 1st controller does not want the bus and forwards RQ to us (RQ->GI->GO->GI)
- 1st controller: three cases: "our RQ is first", "their RQ is first", "at the same time"
a) GI 0->1 is our RQ 0->1: we are the only one or we are first, we take the bus (GRANT-OUT never goes to 1)
b) GI 0->1 is 2nd controller's RQ 0->1: they drove RQ up slightly before we decided we want the bus and drove our RQ up, and it does not matter, we grab the bus, GRANT-OUT never goes to 1, they never see GRANT-IN, they eventually see our RI, drop their RQ, back off and try again.
c) both drive RQ at exactly the same time, this is same as (b), only the 1st controller will see the GI 0->1 transition.

• bus in bad state and timeouts

- RI is stuck at 1: somebody else is driving it, nothing we can do, after a timeout, report an error.
- we drive RQ up, never see GRANT-IN 0->1 transition:
a) RQ is stuck at 1: somebody else is driving it, nothing we can do, after a timeout, report an error.
b) 1st controller: RQ->GRANT-IN jumper is missing, after a timeout, report an error
c) 2nd controller: GO->GI jumper is missing, after a timeout, report an error
d) 2nd controller: 1st controller never gives us the bus, after a timeout, report an error.
- at the end of bus transaction, we want to release RI, but RQ or GRANT-IN are stuck at 1: (we already dropped out RQ-OUT!), somebody else is driving RQ in violation of protocol (they see our RI, they drop their RQ), nothing we can do about it, after a timeout, report an error.
- if the RQ line pull-up resistor is missing, after we release our RQ-OUT (at the beginning of the cycle), the RQ line takes a long time (~6us) to float back to 0. To account for this, the "RQ is stuck at 1" timeout should be sufficiently long.
- whoever is holding RI or RQ up hopefully will eventually timeout and release them. and absent-pull-up-resistor RQ line will eventually float back to 0. therefore, our response to the timeout error should be to try again a few times.

End

AAA