Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis

Objective. The aim of the present work was to develop and test a flexible electrotactile stimulation system to provide real-time feedback to the prosthesis user. The system requirements were to accommodate the capabilities of advanced multi-DOF myoelectric hand prostheses and transmit the feedback variables (proprioception and force) using intuitive coding, with high resolution and after minimal training. Approach. We developed a fully-programmable and integrated electrotactile interface supporting time and space distributed stimulation over custom designed flexible array electrodes. The system implements low-level access to individual stimulation channels as well as a set of high-level mapping functions translating the state of a multi-DoF prosthesis (aperture, grasping force, wrist rotation) into a set of predefined dynamic stimulation profiles. The system was evaluated using discrimination tests employing spatial and frequency coding (10 able-bodied subjects) and dynamic patterns (10 able-bodied and 6 amputee subjects). The outcome measure was the success rate (SR) in discrimination.Main results. The more practical electrode with the common anode configuration performed similarly to the more usual concentric arrangement. The subjects could discriminate six spatial and four frequency levels with SR >90% after a few minutes of training, whereas the performance significantly deteriorated for more levels. The dynamic patterns were intuitive for the subjects, although amputees showed lower SR than able-bodied individuals (86%± 10% versus 99%± 3%). Significance. The tests demonstrated that the system was easy to setup and apply. The design and resolution of the multipad electrode was evaluated. Importantly, the novel dynamic patterns, which were successfully tested, can be superimposed to transmit multiple feedback variables intuitively and simultaneously. This is especially relevant for closing the loop in modern multifunction prostheses. Therefore, the proposed system is convenient for practical applications and can be used to implement sensory perception training and/or closed-loop control of myoelectric prostheses, providing grasping force and proprioceptive feedback.


Introduction
Hand amputation is a dramatic event with long-lasting consequences to everyday life. Myoelectric prostheses can be used to restore the missing motor functions to a limited degree. The user commands the system by activating muscles (e.g., hand and finger flexors/extensors), and the resulting electrical muscle activity is recorded and translated into the control signals for the prosthesis (e.g., close/open) [1]. The control is simple and intuitive since the user employs the muscles that were originally used to accomplish the task. However, surveys show that still approximately 1 in every 5 individuals eventually rejects the prosthesis [2]. An important drawback of active prostheses is that they do not implement somatosensory feedback to the user. Closing the loop in prosthesis control, acknowledged by the users as a relevant future goal [3], might improve the utility and easiness of use (less visual attention) of the system as well as facilitate the feeling of embodiment. These advances are expected to decrease the rejection rates. Furthermore, there are also indications that somatosensory feedback can decrease phantom limb pain [4,5]. Yet, there is only one commerciallyavailable device providing the user with grasping force information through a simple feedback interface (VINCEN-Tevolution 2, Vincent Systems Gmbh, DE) [6].
The missing somatosensory information can be restored using an approach known as sensory substitution [7]. To this aim, the data are collected from sensors in the prosthesis, translated into stimulation profiles and delivered to the sensory structures still present after the amputation. The substitution can be invasive, in the form of a direct nerve stimulation using implantable interfaces [8,9], or noninvasive, within the same modality (e.g., force-to-pressure [10]) or across modalities (e.g., sonic feedback [11]). Direct nerve stimulation is a promising approach, but still in the initial phase of development and far from the actual practical application. The most common non-invasive method to close the loop in prosthetics is to activate the tactile sense by stimulating the skin over the residual limb using direct mechanical (e.g., vibration motors [12][13][14][15][16][17][18], linear pusher [19,20]) or electrical stimulation [21][22][23][24][25][26][27]. In the latter, lowlevel electrical current pulses are delivered to the skin, depolarizing the skin afferents and eliciting tactile sensations. Through the modulation of stimulation parameters, different sensory receptors in the skin can be activated, and thus, the quality and intensity of the elicited sensations can be controlled [1,28]. Therefore, the feedback information can be transmitted to the user by using any combination of pulsewidth, amplitude and pulse rate modulation and/or by changing the active channel (spatial modulation). The latter approach is possible only if a multichannel interface is available.
The grasping force is the most obvious physical variable to be sent to the user since it cannot be determined by visual inspection when grasping stiff objects. This type of feedback was investigated in several previous studies [17,[29][30][31] (see also [32] for review). On the other hand, although it is wellknown that the proprioceptive sense is instrumental for the normal human motor control [33], there are only few studies addressing the methods to provide proprioception to the user [34][35][36]. Furthermore, in most closed-loop prosthetic systems the feedback was provided through a single stimulation channel using amplitude modulation [30,32,[37][38][39]. Multichannel configurations were seldom considered [17,40] and, when implemented, they were realized as an array of individual stimulation units, and not as an integrated interface. For example, Saunders and Vijayakumar [17] used spatial coding to communicate the grasping force of a prosthetic hand via an array of vibrators positioned along the forearm. Witteveen et al [40] evaluated an array of vibrators and electrodes placed longitudinally and circumferentially around the forearm, transmitting the aperture of a virtual hand.
Multichannel interfaces are becoming very relevant with modern developments of advanced hand prostheses, because of the many degrees of freedom (DoF) they can operate. For instance, the Michelangelo Hand by Otto Bock [41] implements hand closing and opening in two grasp types (palmar and lateral) and, if equipped with an active wrist, pronation/ supination and flexion/extension of the wrist joint. BeBionic hand from RSLSteeper [42] and i-Limb [43] from Touch Bionics allow for independent control of individual fingers, and there are research prostheses with up to 16 DoFs [44]. A multichannel feedback interface is necessary to accommodate the flexibility of these systems, since multiple signals need to be transmitted to the user for an effective closed-loop control of the complex devices (e.g., wrist rotation, hand aperture and grasping force). A compact and flexible, fully programmable system, such as the one presented in the current study, can be used in several applications. For example, it can be connected to a prosthesis via a Bluetooth connection (e.g., Michelangelo Hand from Otto Bock) to provide feedback to the user with a good spatial and frequency resolution, thereby closing the loop and facilitating the online control. The resulting closedloop system could represent a test bench to investigate the effectiveness of different coding schemes, such as the dynamic patterns suggested in the present study. Finally, a high resolution interface could prove useful as a sensory discrimination setup, where the user could be trained to discriminate electrotactile stimulation, modulated spatially as well as in parameters. This could be used in therapy of the phantom limb pain [4,5], but also as a general training in perceiving and interpreting the electrotactile feedback in prosthetics.
In this study, we present a novel compact multichannel interface for electrotactile feedback comprising a fully programmable stimulation unit with a custom made firmware and communication protocol connected to an electrode with 16 pads integrated within a single composite flexible substrate. The interface was evaluated by conducting psychometric tests in able-bodied and amputee subjects, using two electrode types, and employing spatial and frequency coding, as well as a set of dynamic stimulation patterns designed to transmit the grasping force and proprioceptive information from a multi-DoF prosthesis. The tests demonstrated that the novel interface allows simple control of electrotactile stimuli location and stimulation parameters, the two electrode types resulted in a similar performance, and the three experiments we performed showed that the subjects could reliably (>90%) discriminate between 6 spatial locations, 4 frequency levels and 8 dynamic patterns after a minimal training. To the best of our knowledge, this is the first evaluation of a potential set of patterns for transmitting the state of a multi-DoF prosthesis back to the user.

Novel electrotactile interface
Figure 1 depicts a novel system for multichannel electrotactile stimulation which can be used either as a standalone unit for the perception training, as in sensory discrimination treatment [4], or connected to a multi-DoF prosthesis to convey the information about the system state, thus closing the loop in prosthesis control. The system comprises a single channel stimulator, multiplexing unit and custom-designed multi-pad electrodes for electrocutaneous stimulation (figure 1, left panel). The stimulator implements a user interface for the manual adjustments of the stimulation parameters through a touch screen as well as a graphical visualization of the states of the electrode pads (active/inactive). The stimulation parameters (channel state, pulse amplitude, pulse width) can be adjusted independently and individually for each channel, thereby enabling flexible implementation of different unimodal as well as mixed coding schemes (e.g., simultaneous change in location and frequency). The only common parameter is the stimulation frequency, which is set globally for all the channels. A key feature of the stimulator firmware is a multi-DoF feedback mode for automatic mapping of the input signals describing the state of the prosthesis to the parameters of electrical stimulation. The prosthesis state can be transmitted to the user using custom-designed protocols, including frequency, intensity and/or spatial coding as well as a set of standard preprogrammed dynamic patterns (see section 2.2) communicating the proprioceptive and grasping force information.
The system includes a stimulation unit (MaxSens, Tecnalia, San Sebastian, Spain) specifically designed for the requirements of the electrotactile stimulation, i.e., generating current-controlled biphasic stimulation pulses with a pulse width from 50 to 1000 μs (10 μs step), pulse rate between 1 and 400 Hz (1 Hz step) and amplitude in the range of 0.1-5 mA (0.1 mA step). The stimulator allows activation of 16 independent electrode pads using a multiplexing unit to distribute the pulses generated by a single stimulation channel. The block scheme of the proposed system is presented in figure 2. Stimulation pulses for all active pads on the multipad electrode are time distributed within one stimulation period and forwarded to the multiplexing unit according to the order of activation. The multiplexing unit routes the stimulation pulses through 16 solid-state relays to distribute the stimuli to the electrode pads based on the activation schedule. The activation schedule defines the mapping between the pulse number and the electrode to which that pulse will be delivered. As an example, figure 2 depicts two pulses of different amplitude that are forwarded to electrode pads 1 and 16. MaxSens allows online control of the stimulation parameters, including the channel state (on/off), via a Bluetooth communication interface. To this aim, a simple communication protocol was implemented comprising textual commands that can be sent from any device (e.g., host PC, dedicated controller or prosthetic hand). The communication protocol allows read out and setting of the parameters for each channel individually using low-level messages, but it also includes high-level commands that can set several stimulation parameters at the same time when the stimulator is working in the feedback mode. The high level command comprises the information about the state of a multi-DoF prosthesis as read from the prosthesis sensors (i.e., the current aperture, wrist rotation, wrist flexion/extension, grasping force) and the stimulator implements the protocol to map the received sensor data to the stimulation profiles. The examples of high-level commands are the dynamic patterns explained later (see section 2.2). These high level commands could be used with Michelangelo Hand from Otto Bock, which has an embedded Bluetooth transceiver that would enable the stimulator to connect to the hand directly. The prosthesis integrates sensors that measure the position of the joints and the applied force. Measured sensor data can be sent to the stimulator, which would directly map the current state of the 4 DoFs of the prosthesis to the proposed dynamic stimulation patterns. In the future step, such a feedback system can be miniaturized so that the electrode and the stimulator electronics are embedded into the prosthetic socket. For the use in the present experiments we have developed a simple user friendly PC application that provides graphical interface for controlling all the stimulation parameters of interest through low-level and high-level commands.
The stimulation electrodes were made of a polyester layer, an Ag/AgCl conductive layer, and an insulation coating covering the conductive leads. To improve the electrodeskin contact, conductive hydrogel pads (AG702, Axelgaard, DK) were placed on the top of the electrode pads. This design allowed electrode bending and provided a close contact of the electrode with the skin. To match the maximum number of outputs in the multiplexing unit, we designed electrodes with 16 pads. Two types of 1×16 multi-pad electrodes were produced in different sizes and shapes, based on the average forearm circumference (figure 3). The first electrode system featured 16 circle-shaped cathodes with a long common anode stretching alongside (common anode configuration, CAC). The second system consisted of 16 concentric cathodeanode pairs (concentric electrode configuration, CEC). We designed these two electrode types aiming to investigate the advantages in spatial discrimination that concentric configuration might provide due to more localized and superficial current flow. The size of the electrodes and the center to center distance between pads (denoted on figure 3) were  The electrodes were designed to be placed circumferentially around the forearm, since this orientation provides better localization than longitudinal placement [23,46]. Electrodes were taped to the flexible Tennis elbow support from Mueller (Mueller Sports Medicine, Prairie du Sac, United States), that ensured easy placement and positioning of the electrode with a secure adjustable strap. A multi-pad electrode positioned on the forearm of an amputee and of an able-bodied test subject is shown on the right panel of figure 1.

Dynamic stimulation patterns
A set of stimulation patterns were defined and programmed in the firmware of the stimulation unit in order to intuitively communicate the states of a multi-DoF prosthesis. In total, eight patterns (table 1) were defined to represent the activation of four prosthesis variables (aperture, grasping force, two wrist rotations) in two directions (opening/closing, force increase/decrease, wrist pronation/supination, and wrist Table 1. Dynamic stimulation patterns communicating the states (proprioception and force) of a myoelectric prosthesis.

Degree of Freedom Graphical representation Description
Hand opening/closing Hand opening-activating two neighboring pads starting in the middle of the electrode (hand fully closed) and moving circumferentially and further away from each other until the ends of the electrode (hand fully opened). Hand closing-the same pattern, with opposite starting position and direction of the movement of the active pads.

Wrist Pronation/ supination
Activating two neighboring pads starting at the center of the electrode and moving together circumferentially in the direction of hand rotation (counter-clockwise or clockwise, depending on the amputation side) until the end of the electrode.

Grasping force increase/decrease
Modulating the stimulation frequency of the two central electrode pads through the four discrete frequency levels, from lowest to highest for force increase and opposite for the decrease.
flexion/extension, respectively). The selected set accommodates the capabilities of the Michelangelo Hand from Otto Bock, as an example of application, but other choices may be used to match with the characteristics of other prosthetic hands. In order for the patterns to be intuitive, the design principle was to represent the spatial variables, such as pronation/supination and aperture, using spatial coding, and the intensity variables (e.g., grasping force) with parameter modulation (frequency). Furthermore, the spatial codes were designed so that they resemble the movement performed by the prosthesis. For instance, hand closing is represented by moving two active pads closer together, thus resembling the movement of the fingers, and hand rotation is represented by the rotation of two active pads in the same direction. We did not consider intensity coding, which includes amplitude or pulse-width modulation, since there is a substantial variability in the dynamic ranges across subjects. For example, some subjects are sensitive to electrical stimulation and can withstand only low amplitudes/pulse widths, resulting in a narrow range of stimulation intensities available for the information coding. Finally, in each pattern at least two electrode pads were active at any given moment (table 1). This was adopted due to the fact that dual-channel stimulation results in more reliable perception thresholds and sensory feedback codes compared to a classic single channel approach [47]. The frequency levels for the pattern communicating the grasping force corresponded to the ones obtained in the frequency discrimination test (see section 2.3.3). The intensity (amplitude and pulse width) was adjusted for each electrode individually so that the elicited sensations were clear, welllocalized but not uncomfortable. Each pattern was designed to represent a single prosthesis variable in an intuitive manner. However, considering the future practical applications, the patterns were constructed so that they can be combined with one another, providing thereby feedback regarding two or more of the prosthesis states. For example, as the user closes the hand, the two electrodes come close together (aperture pattern). If the user then starts rotating the wrist, the two electrodes would start rotating around the forearm (rotation pattern) while maintaining the relative position (constant aperture). Another example is when the hand squeezes a compliant object. The user will be able to detect both the change in the applied force through parameter modulation (force pattern) and the deformation of the object through spatial coding (aperture pattern). Importantly, the messages for aperture, wrist rotation and grasping force are also intended to communicate the magnitude of the corresponding variable (and not only its identity). For example, the user will be able to determine the current aperture or wrist rotation angle by recognizing the currently active electrode pads, i.e. spatial coding of the magnitude in the ongoing stimulation pattern. In addition to the absolute information regarding the state of the hand (e.g. hand halfclosed), the user can also determine the rate of change (e.g. opening at certain velocity) from the proposed dynamic patterns. In other words, the patterns were indeed designed to continuously transmit the present state of the prosthesis to allow for the online closed-loop control.
The patterns for the wrist flexion/extension were the only patterns coded to represent just the identity of the movement (i.e., the wrist is flexing/extending), without providing the information on the movement amplitude (flexion or extension angle). They were included in the tests for completeness, but they were deemed to be less important for the future practical application since there are yet no commercially available prostheses with an active flexion/extension of the wrist.

Psychometric tests
The purpose of the electrotactile interface is to transmit the information to the user reliably, consistently and with a suitable resolution. To evaluate the developed system, we conducted three psychometric tests investigating the quality of information transmission using spatial and frequency coding using different resolutions (i.e., number of levels) and two types of array electrodes. In the final experiment, we tested the subject ability to discriminate the proposed dynamic stimulation patterns. The experiments in able-bodied subjects were conducted in Tecnalia Serbia and the test in amputees was performed at the Specialized Hospital for Rehabilitation and Orthopedic Prosthetics, Belgrade, Serbia. The experiments in able-bodied subjects included 11 healthy volunteers. Nine subjects participated in all the tests. One subject who took part in the assessment of the spatial and frequency discrimination was unable to participate in the test with the dynamic stimulation patterns. Therefore, an additional ablebodied subject was recruited for this test as a replacement. The experiments in able-bodied subjects were performed at the same location using identical setup.
2.3.1. System setup and calibration. The system setup and electrode calibration were performed for each subject and each session, to accommodate subject-specific sensitivity and repositioning of the electrodes across sessions. Subjects were comfortably seated in front of a table. The electrode was placed around the subject's right forearm, midway between the elbow and wrist, and the stimulation unit was placed on the table next to the forearm. The stimulation intensity was adjusted for each pad on the electrode used in the experiment, based on subjectively perceived sensations. Pulse width was set to 200 μs and the sensation threshold was determined using the method of limits [48] by modulating the current amplitude. This step was performed semi-automatically using the host PC application for automatic calibration. The experimenter configured the parameters (e.g., pulse width, rate, stimulus duration) and started the test. The program automatically increased the stimulation amplitude by 0.1 mA steps until the subject reported that he/she felt the sensation. The final amplitude for a specific pad was set above the threshold in order to obtain pleasant, but distinct sensation. The experimenter then used the PC application to fine-tune the stimulation amplitudes across the pads so that the sensations of similar intensity are perceived by the subject for each pad of the electrode. Thanks to this, system setup and calibration process almost never required more than 5 min, which is significant asset of the presented system, having in mind that with the conventional electrical stimulators only the placement of 17 electrode pads would likely take more time.
2.3.2. Spatial discrimination. The test was performed in 10 able-bodied volunteers (5 male, 5 female, 29±9 yrs) who signed an informed consent form for the study that was approved by the local ethics committee. The stimulation frequency in this experiment was fixed to 100 Hz. The aim was to investigate how the success rate (SR) in localizing an active pad depends on the spatial resolution (number of utilized electrode pads). Three spatial configurations were tested for the CAC and CEC electrode types (figure 3): (1) Four pads with three-pad gap in-between (pads 3, 7, 11, and 15) (2) Six pads with two-pad gap in-between (pads 1, 4, 7, 10, 13, and 16) (3) Eight pads with a single-pad gap in-between (pads 1, 3, 5, 7, 9, 11, 13, and 15).
The experiment consisted of training and testing session that were repeated for each of the three configurations in a randomized order using two types of flexible electrodes (CEC and CAC). The training session included 40 trials. In each trial, one of the pads from the tested configuration was activated for 3 s and visual feedback was presented to the subjects, i.e. the active pad was indicated through the blinking of the corresponding LED indicator on the graphical representation of the multi-pad electrode (figure 4). During the training, the subjects received the stimulation only through the electrode pads that were used in the specific test. The pads were activated successively (i.e., from lower to higher numbered pad, see figure 4) because the goal was to train the subjects to discriminate successfully in the most difficult case, which is the stimulation delivered through two neighboring pads. Each trial was followed by a 1 s break. In the testing session, which also included 40 trials, the subjects were asked to identify one of the randomly activated pads without visual feedback. During testing, the stimulation lasted until the subject provided an answer and there was 1 s break between the trials. The spatial discrimination test lasted approximately 60 min.
We also tested the spatial coding in one able bodied subject (27 yrs) using a prolonged training protocol. The aim was to test if the subject could learn to utilize the full resolution of the 16-pad interface. In this test, the trials were presented continuously while the SR was evaluated based on the last 40 trials, and the experiment was terminated once the SR was maintained over 90% for 10 consecutive trials. In each trial, the subject was asked to identify one (out of 16) randomly activated pad, and was subsequently presented with the correct answer. The stimulation lasted until the subject provided an answer, followed by a 1 s break. The trials were interrupted with a longer break on request from the subject.

Frequency discrimination.
The aim was to evaluate how the SR in recognizing a set of frequencies depends on the effective resolution (number of utilized frequency levels). The test was performed in the same subjects, following the assessment of the spatial modulation, using the CAC electrode type (figure 3, left). Only one electrode type was tested since the perception of the frequency should not depend substantially on the pad design, but rather on the intensity of stimulation, i.e., with increasing intensity, it becomes easier to perceive the pulses and thereby the changes in their rate of delivery. It was therefore assumed that if the same intensity would be set on both electrode types, which might require different pulse amplitude due to a different pad design, the results in frequency discrimination test would be similar. The frequencies were selected within the range of interest (between 4 and 100 Hz) based on just noticeable differences (JNDs), as explained below, so that the chosen values were maximally separated in the perceptual space. The upper limit of 100 Hz was selected because the stimulation at higher frequencies elicits a fused sensation of tingling, which becomes similar across frequencies and therefore hard to discriminate [49], and the lower limit of 4 Hz was chosen considering the dynamics of the daily life activities using the prosthesis.
The JNDs were estimated in five able-bodied subjects (27±4 yrs) using the method of manual adjustments [48]. It should be noted that there are more advanced methods (e.g., adaptive staircase procedures) to determine the JNDs, which are less susceptible to subjective factors compared to the method of adjustments used in the present study. However, a simple approach was selected for this test due to time constraints, since this was only a preparatory step for the main experiment (frequency discrimination). Most importantly, the Webber fractions that were obtained agree with the results reported in literature. The setup and calibration were identical as when evaluating the spatial modulation. Two central pads (8 and 9) activated simultaneously were used as the stimulation point. Two stimuli were delivered to the subject in succession (3 s first stimulus, 1 s break, 3 s second stimulus) and repeatedly (1 s break): a current baseline frequency ( F i ) and the baseline plus an increment ( F i +ΔF ), where ΔF was manually increased until the subject reported that he/she could sense the difference between the two stimuli. The resulting ΔF was adopted as the JND for the current baseline, and the test was repeated assuming that the next baseline was the current baseline plus JND ( F i +JND). The procedure continued until reaching the end of the frequency range ( F i >100 Hz). The result was a sequence of frequency levels F k , where k=1, 2 KN is the step number and F k and F k−1 were separated by a JND. We pooled the data from all subjects and fitted a quadratic function F(k), using F k and k as the dependent and independent variables, respectively. The frequency levels were then selected by dividing the k-scale equidistantly using the desired number of points k i and by reading off the corresponding frequencies from the fitted model ( F(k i )). Four sets of frequencies including 3, 4, 5 and 6 levels, respectively, were selected in this manner. All four sets were evaluated in random order in each subject.
The training session comprised 40 trials. Two central pads were activated at a frequency from the current set and visual feedback was provided to the subjects as a blinking LED indicator (figure 4, right). The frequencies were presented successively, from lower to higher, and repeatedly, starting again from the lowest frequency when all the frequencies from the set were presented. The stimulus lasted for 3 s and there was 1 s break between the successive stimuli. During testing, comprising 40 trials, the subjects were asked to identify one of the randomly activated stimulation frequency levels. In each trial, the stimulation lasted until the subject provided an answer, and there was a 1 s pause between the trials. The frequency discrimination test lasted approximately 60 min.
2.3.4. Discrimination of dynamic stimulation patterns. The aim of this test was to evaluate how well the subjects can discriminate between a set of dynamic stimulation patterns communicating the state (artificial proprioception and force) of a multi-DoF prosthesis. Dynamic patterns rely on spatial and frequency coding. The results demonstrated that both electrode types lead to similar performance in spatial discrimination and, as explained before, the electrode design per se should not affect the SR in frequency discrimination. Therefore, it is likely that both electrode types would result in similar performance when recognizing the dynamic patterns. The test was therefore performed using only the CAC electrode (figure 3, left) in 10 able-bodied volunteers (5 male, 5 female, 31±9 yrs) and 6 amputees (5 male, 1 female, 46±6 yrs). The experimental protocol was approved by the ethics committee of the Specialized Hospital for Rehabilitation and Orthopaedic Prosthetics Belgrade, Serbia. All participants signed an informed consent form and provided permission for the publication of photographs for scientific purposes.
The demographic data for the amputee subjects who took part in the dynamic stimulation patterns experiment is presented in table 2. All participants used only passive (cosmetic) prostheses and had no previous experience in myoelectric control. For every subject, except subject ID5 (bilateral amputation), the garment with the stimulation electrodes was positioned 5 cm distal to the elbow on the amputated limb, and the electrode was centered to the central volar region. For the subject ID5, the experiment was performed on the left upper arm, 5 cm proximal to the elbow, and the electrode was centered over the biceps brachii, since the residual limb was very short. The setup was the same as in the test for the spatial and frequency discrimination.
The patterns were trained one by one, in a randomized order, by presenting the two patterns coding the same DoF in two different directions (e.g., hand opening and closing). For simplicity and ease for the subjects, the channel stimulation timing in the dynamic patterns was adopted to last for 2 s in total, which corresponds to slow hand movements. As an example, for the hand closing pattern, each electrode was active for 0.25 s so that the total time for the stimulus to travel around the forearm, from the initial to the final position, was exactly 2 s. The subject was asked to identify the presented pattern and the experimenter provided verbal feedback about the correct answer. This was repeated until the subject became confident in differentiating the two patterns, after which the training proceeded to a next DoF. The training was organized like this based on the assumption that the dynamic patterns were intuitive, naturally related to the prosthesis variables that they coded, as explained before. It was expected that for the subjects, it would be the easiest to learn the patterns representing different directions of the same DoF first, since  Trauma Transradial  25  ID2  F  46  4  Right  Right  Trauma Transradial  20  ID3  M  45  1  Right  Left  Trauma Transradial  27  ID4  M  43  5  Right  Right  Trauma Transradial  23  ID5  M  59  39  Right  Both  Trauma Transradial left  14  Transhumeral  right  ID6  M  42  5  Right  Right  Trauma Transradial  24 in this case only the direction of the spatial (parameter) modulation was different. During the training, the subjects were also asked to visualize the movement of the prosthesis that each pattern represents and to associate that movement to the presented stimuli. Therefore, the training was organized accordingly. The subjects were trained to discriminate the two patterns for the same DoF using reinforced learning. The subjects then proceeded to the next DoF, until all the DoFs were trained once. It was assumed that the subjects would memorize the patterns across the DoFs with no major difficulties, and that repeating the training for the same DoF several times would not be necessary. This was indeed confirmed by the study results. In total, the training lasted for approximately 20 min. In the following testing phase, which comprised 40 trials, the subjects were asked to identify a randomly activated stimulation pattern out of 8 possible patterns, without feedback from the experimenter.

Data analysis.
The outcome measure in all the tests was SR (%) in correctly identifying the information that was transmitted through the electrotactile interface. For frequency discrimination, the statistically significant differences in the mean SR within a group of tested conditions were evaluated using one-way ANOVA and Tukey's honestly significant difference criterion for the post-hoc pairwise comparisons. For spatial discrimination, two-way ANOVA was performed with the electrode types (CAC versus CEC) and the number of active pads (four, six and eight) as the factors. For both frequency and spatial discrimination, Levene multi-sample dispersion test was performed to determine if there were statistically significant differences in variances between the conditions. If the test indicated significance, pairwise comparison was performed using F-test for equal variances, with Bonferroni correction to account for multiple comparisons. The difference in the discrimination of dynamic stimulation patterns between able-bodied subjects and amputees was tested for statistical significance using t-test for means, and F-test for variances. Also, one-way ANOVA was performed to determine whether there were significant differences in the mean SR across amputated subjects and the patterns. The threshold for the statistical significance was set at p<0.05. The results in the text are reported as mean±standard deviation. For the dynamic pattern discrimination test performed on amputees we have also provided confusion matrices to better describe the misinterpretations between the proposed patterns.

Spatial modulation
The bar plot of the average SRs obtained across spatial resolutions of the two tested electrode designs are presented in figure 5. Spatial resolution (implying number of active pads and their configuration) was the only significant factor (p<0.001). Electrode type was non-significant (p=0.4085), nor the interaction (p=0.5873). The average SR for the CAC electrode ( figure 3, left) was 98%±3%, 88%±11% and 66%±24% when four, six and eight pads were used, respectively. For the CEC electrode (figure 3, right), the mean values for four, six and eight pads were 97%±5%, 90%±13% and 74%±15%, respectively. For both electrode types, the mean SR with four and six pads was above 90% and there was no statistically significant difference between these two conditions. With eight pads, however, the performance deteriorated considerably. Considering the inter-subject variability, the variance for four pads was significantly (p<0.001) lower compared to that obtained for six and eight pads, for both CAC and CEC electrode. The variance between the two electrode types for the same number of pads was similar (no significant differences).
The results of the prolonged training experiment for one able-bodied volunteer are presented as a bar plot in figure 6. The training included 440 trials and lasted approximately 2.5 h. Each bar plot presents SR in recognizing 1 out of 16 possible stimuli locations averaged across 40 trials. Initially, there was no increase in SR, but after some time the subject started to relate the subjectively perceived differences in sensation to the particular stimuli location and, in the second half of the training the performance steadily increased.

Frequency modulation
The average Weber fraction, defined as the JND normalized by the baseline, for the frequency range of 4 to 100 Hz was 0.23±0.1, which is in accordance with the results reported in literature [50,51]. The average number of frequency levels separated by the JNDs was 15±2. The graphical representation of the results obtained in the JND experiment along with a quadratic function used as a mathematical model to describe the nonlinearity in the frequency discrimination is presented in figure 5. Based on this mathematical model, the following sets of frequencies were selected: (1) Three frequency levels-4, 27, 100 Hz The results of the frequency discrimination experiment are presented using a bar plot in figure 8. Average SR was 99%±1%, 95%±8%, 80%±16%, and 74%±17% for three, four, five and six levels, respectively. The SRs for three and four levels were similar and higher than 90% and significantly better compared to SRs obtained for five and six levels. There was no statistically significant difference between the last two conditions. The inter-subject variability for the levels from 4 to 8 was similar (no significant differences) and substantially higher compared to the variability for the three levels (p<0.001).

Discrimination of dynamic stimulation patterns
The average SR for amputees and able-bodied subjects when discriminating the eight dynamic stimulation patterns transmitting the state of the four prosthesis DoFs is presented in  The JND curve, describing relation between the frequency of stimulation and the number of levels that subjects reported they perceived as different sensations. Number of levels that each of the five subjects could identify in the frequency range from 1 to 100 Hz is presented on ordinate to emphasize the decrease in sensitivity on higher stimulation frequencies. The quadratic function fitted based on this data and used as a model for selecting frequency levels for frequency discrimination tests is plotted using a full line.  Table 3 holds SR percentage for the proposed dynamic stimulation patterns for each of the six amputees and gives the average SR per amputee and average SR per pattern. Able-bodied subjects were able to perceive and identify stimulation patterns with an average SR of 99%±3%. The average SR for amputees was 86%±10%, and the difference was significant both in the mean (lower, p=0.01) and the inter-subject variability (higher, p<0.001) compared to the able-bodied subjects. Table 3 reports the SRs per subject and pattern, as well as the average results. Figure 10 shows confusion matrices characterizing the discrimination of stimulation patterns in individual amputee subjects (ID1-ID6) as well as the confusion matrix for the average performance across all 6 subjects. All the subjects were able to recognize at least four dynamic stimulation patterns with the SR of 100%, but there was not a single pattern that was recognized with the SR of 100% in all the subjects. This implies that individual perception abilities played a major role in the pattern misclassification (table 3). The highest mean SRs were obtained for the stimulation patterns correlated to force increase and decrease (93% and 98%, respectively). The subjects made the most errors when presented with the stimulation patterns for hand opening and closing, with a mean SR of 74% and 83%, respectively. These patterns were commonly misinterpreted as wrist pronation/ supination, and vice versa. Subjects ID1, ID3, and ID6 were the most successful in pattern recognition, with an average SR above 90%. Subject ID2 had the most difficulties to recognize hand closing pattern (SR of 29%), subject ID4 hand opening (SR of 20%), and subject ID5 wrist pronation (SR of 50%). However, there was no statistically significant differences in the SRs across subjects (p=0.399) nor across the stimulation patterns (p=0.720).

Discussion
We have designed and tested a novel, compact and fully programmable system for multichannel electrotactile stimulation. The system includes a custom designed hardware, Figure 8. Success rate of stimulation frequency recognition for three, four, five and six different frequency levels, tested on 10 able-bodied subjects using CAC electrode. Horizontal bar with asterisks indicates the statistically significant difference in mean SR between the respective conditions (numbers of frequency levels). ( * , p<0.05; ** , p<0.01; *** p<0.001). Figure 9. Success rate in discriminating the dynamic stimulation patterns in able-bodied volunteers and amputees. Able-bodied subjects were able to discriminate the stimulation patterns with no difficulties, while the performance for amputees was lower in average and more variable. Horizontal bar with asterisks indicates the statistically significant difference in the mean. ( * , p<0.05; ** , p<0.01; *** , p<0.001).  ID1  100  100  75  100  100  100  100  100  97±9  ID2  75  67  100  100  29  60  100  100  79±26  ID3  100  100  100  100  100  100  80  60  93±15  ID4  100  67  83  100  100  20  100  90  83±28  ID5  100  100  100  86  67  67  50  100  84±20  ID6  71  100  100  100  100  100  100  71 93±14 Average for all subjects (MEAN±STD) firmware and a flexible array electrode placed circumferentially around the forearm/residual limb. The advantage of the developed interface with respect to conventional electrotactile systems is that this is an integrated solution. The flexible electrode comprises 16 pads which are all positioned at once by strapping the electrode to the forearm, as explained before. This is in contrast to conventional systems [52], where the electrodes need to be placed one by one. In addition, due to external connection and programmability of the interface, the stimulator is controlled through a user friendly PC application with automatic calibration. We evaluated the system using two types of electrodes (CAC versus CEC) and psychometric tests performed in able-bodied and amputee subjects. The spatial discrimination tests (figure 7) indicated that there was no significant difference in the quality of discrimination between the two electrode types. We expected that the concentric electrodes might facilitate the recognition due to a localized current flow. However, CAC electrode was designed to have the anode very close to the cathode pads, which likely resulted in a similar localization effect as the for the concentric ring. Concentric electrode design, corresponding to CEC electrode, is traditionally recommended for the application of the electrotactile stimulation [26], but the full pad electrode design is being used as well [40,53] and the studies evaluating objectively the advantages/disadvantages of the two configurations are rare. Higashiyama et al [23] showed that the average errors of localization for electrodes positioned transversally across the forearm was similar for full pad and concentric electrodes.
The TPTD for electrotactile stimulation was investigated comprehensively in [45]. The study reported that TPDT is a function of the stimulation intensity (pulse width and amplitude), frequency and the body location. The tests were conducted at a range of frequencies (5-100 Hz) and for three different pulse widths (10 μs, 100 μs and 1 ms) demonstrating that the best results (minimal TPDT ∼9 mm) was achieved for frequencies below 50 Hz and the pulse width of 100 μs. The stimulation intensity was set to just below the pain threshold (strong stimulation). In the present study, the pulse amplitude was set to be slightly above the sensation threshold (low intensity) to minimize the current spread and thereby provide a more localized sensation. The pulse width was 200 μs, which is close to the optimum reported in [45], and the stimulation frequency in spatial discrimination tests was set to 100 Hz to provide clear sensation [54]. Therefore, the interpad distance for CAC and CEC electrodes of 12 mm and 13.5 mm, respectively assured that the subjects could discriminate the activation of the neighboring pads. The latter was additionally confirmed by the pilot tests. The present study therefore complements the previous results [23] by demonstrating that other configurations, which might be more convenient for production and application, can lead to a similar performance when using spatial coding as the commonly used concentric solution. Here, it should also be noted, that based on the results of spatial discrimination experiment we chose to use only one electrode type in the subsequent dynamic pattern recognition test, having in mind that dynamic patterns rely solely on the spatial and frequency discrimination.
The spatial and frequency discrimination tests aimed to assess the effective resolution of the feedback presented through these two parameters of electrical stimulation. It should be noted that this effective resolution represents the number of stimuli locations and frequency levels that the subjects can recognize and correctly identify, when they are provided randomly. In this case, the subject needs to memorize a set of N stimuli so that he/she can guess the stimulus selected randomly from the set (absolute recognition). This is different from the total number of stimuli that the subject can tell apart when they are presented in pairs of two (relative discrimination), as measured through TPDT and JND, respectively. It can be expected that the subject will be able to discriminate more stimuli relatively then absolutely. However, the latter is important for the practical application since it provides complete information (e.g., not only that the variable, prosthesis force, has changed but also to which level).
After only a short training (∼4 min), able-bodied subjects were able to discriminate six spatial positions and four frequency levels with SR above 90%, and with no statistically significant decrease in performance compared to the lower resolutions with maximal identification rates (four spatial and three frequency levels). If the two coding schemes are combined, the total resolution of the feedback interface could become 24 (6 pads×4 frequencies per pad). Since the two coding schemes are perceptually separated (spatial localization versus frequency perception), it is likely that the SR for the mixed coding including combined spatial and frequency modulation would not be substantially different from that obtained for the single coding, i.e., spatial only or frequency only. We base this assumption on the results from our recent study [55], where mixed spatial (5 levels) and frequency (3 levels) coding was tested and the overall SR for the 15 codes (5×3) was 87%, with the marginal SR of 93% and 95% for the spatial and frequency levels. This was similar SR to that obtained for 6 spatial and 3 frequency levels in the present study (88% and 99%, respectively). This implies that recognizing two parameters simultaneously (location and frequency) is not substantially different from recognizing each parameter individually. Nevertheless, this is a hypothesis that needs to be tested in a dedicated experiment. The mixed coding interface could be therefore used to transmit high resolution information (24 levels) about the variable of interest (e.g., grasping force) during closed-loop sensory training or myoelectric prosthesis control. However, this still needs to be confirmed experimentally.
There are only few studies that evaluated the discrimination of multiple spatial channels [40,56] and frequency levels [54]. In [45], the SR in recognizing five electrotactile stimulation channels placed with an inter-channel distance (center to center) of 4 cm was 94%±9%, which is in accordance with our results , i.e., the SR for five channels in [45] is in-between the SR for four and six channels obtained in the present study. Importantly, the present experiment also demonstrated that the performance decreased substantially if the electrode number was further increased to eight channels. Frequency discrimination was addressed in [54] using wire electrodes placed close to the nerve demonstrating that a trained subject could discriminate up to six discrete levels with an SR higher than 75%, which is also in line with our results using surface electrotactile stimulation. All in all, the present study determined the maximum limits for the resolution of the spatial and frequency modulations that can be accommodated by the subjects after few minutes of training. In addition, the results provide concrete guidelines for frequency level selection based on JND tests and electrode pad positions to achieve the reported performance. This is an important information when designing the sensory training and closed-loop myoelectric control systems with an explicit reliable feedback (high SR) and minimal start-up time (mount, configure, train and run).
The result of the experiment with the prolonged training is encouraging, since it demonstrated that an able-bodied subject can reach high performance (SR90%) in recognizing all 16 pads after two hours of reinforcement learning. Therefore, the full spatial resolution of the developed interface can be utilized with a proper training. Similar increase in performance after prolonged training might be expected for the frequency discrimination, as demonstrated by Riso et al [52,57]. Taking into account the training effect, we could envision an application in which the user could initially start with a limited number of spatial×frequency levels (e.g., 24, as explained above), and then increase the resolution as he/ she becomes more familiar with the interface and the elicited sensations. The limiting factor for recognizing two different stimuli locations is defined through TPDT and for identifying stimulation frequency by the JNDs in the frequency range of interest. This leads us to the hypothesis that up to 16 spa-tial×15 frequency levels could be utilized by supremely trained subject. However, whether somebody can be trained this well and how much time is required to achieve this performance remains to be tested.
All but one able-bodied subject were able to recognize the dynamic stimulation patterns communicating the prosthesis states without making a single mistake. This was possible after only a brief training, demonstrating that the selected patterns were intuitive and easily discriminable. We expected to obtain similar results in amputee subjects. The SR was indeed high (86% overall average), but still lower compared to able-bodied subjects. The decrease in performance might be due to the fact that the amputation affects the sensory capacities of the residual limb and/or changes the geometry of the sensory-motor structures, but it can also be the result of the difference in educational level, general experience with technology, etc between the able-bodied subjects (mainly composed of graduates of electrical engineering) and amputees who participated in experiments. Nevertheless, the result of the test in amputees is an encouraging outcome since the performance was similar across patterns and subjects (no statistically significant differences). This demonstrates that the amputees were able to consistently discriminate a set of presented dynamic patterns despite the diversity of the individual characteristics, such as, amputation level, type of trauma, surgical procedure, time elapsed after the procedure, overall physical and psychological condition of the subject, and even the location of the electrotactile feedback. With respect to potential practical application in closed-loop prosthetics, the subjects were able to discriminate the active state of the prosthesis as well as the direction of change with high SR by relying on the set of designed stimulation patterns. However, there is an indication that the similarity between the patterns could affect their interpretation, as in the case of the hand closing/opening and wrist pronation/supination in some subjects (ID2 and ID4). Both patterns included a stimulus moving transversally around the forearm. This ambiguity could likely be decreased with a longer training. The future tests will evaluate this aspect and also test if the subjects would be able to interpret not only the state and direction of change, but also the current level of the prosthesis variable (proportional decoding).
The high SR in recognizing the dynamic patterns by the amputees receiving only a brief training is an encouraging result. This implies that the prospective prosthesis users would be able to perceive and interpret the electrotactile codes and that the dynamic patterns can be therefore used to provide somatosensory feedback from the prosthesis. Importantly, as explained before, the patterns were carefully designed to be intuitive and to allow superposition, in order to provide simultaneous feedback on multiple prosthesis variables. When connected to a prosthesis, the presented feedback interface will be, to the best of our knowledge, the first system capable of transmitting multivariable feedback about the full state of an advanced multifunction prosthesis (e.g., Michelangelo Hand). Therefore, the feedback system and the methods presented in the current study could potentially bring a substantial benefit to the prosthesis users. They could allow more intuitive and effective control of advanced prostheses (which are known to be challenging to operate [58]). However, this is only a hypothesis that needs to be thoroughly tested in the future experiments. Importantly, the electrotactile system and the flexible electrode seem to be robust and reliable. All the evaluations in the present study, together with the preceding pilots, have been done using the same stimulator and electrode (no malfunction or damage). Our objective is to further improve the practicality of the system, miniaturize the electronics to integrate the stimulation system and the electrode into the socket of the prosthesis and, thus, develop a self-contained prosthesis with an advanced feedback to the user.
The setup of the system and the calibration were simple and fast due to the overall system design. The electrode needs to be wrapped around the forearm and strapped, and the calibration is performed semi-automatically using a PC program. In the present study, the experimenter led this process, but it is likely that this simple procedure (∼5 min) could be easily executed by the prosthesis user alone. Basically, the user only needs to adjust the stimulation amplitude to just above the sensation threshold. In the future, this could be done even without a PC application (used in the present study). As explained before, the future goal is to integrate the system within the socket. Once the user mounts the prosthesis, the calibration could be activated through a dedicated button on the socket. The stimulator would start increasing the pulse amplitude and the user would indicate that he/she feels the sensation by pressing the button. Alternatively, a muscle contraction could be used as the command interface. During repeated donning, the sensations will change due to repositioning of the electrodes. However, it is likely that the user would need to recalibrate the system only occasionally, since the only requirement is that the stimulation is above the threshold. Initially, the user would need to train the patterns, which can be done similarly as in the present study. The first phase would be psychometric training, followed by the functional use of the prosthesis with the feedback activated to support the online control.
In the present study, the dynamic patterns have been tested psychometrically, which can only provide insights into the intuitiveness and understandability of the proposed coding schemes. The real practicality of the dynamic patterns needs to be tested in the closed-loop conditions, where the subjects would operate a prosthesis using electrotactile feedback. The present study evaluated the stimulation interface in open-loop conditions, where the user passively received the electrotactile codes in order to test discrimination capability. This is a different context from the intended real life application in which the task for the user will be to control the prosthesis based on the online feedback. Therefore, the next important step in this research is to evaluate the dynamic stimulation patterns in functional tests with a real hand prosthesis (e.g., Michelangelo hand from Otto Bock). The subjective perception of the stimulation patterns might be different in the closed-loop conditions. First of all, the task is more complex, since the user will have to simultaneously generate prosthesis commands (muscle activation) and monitor the feedback. This might require increased attention and cognitive load, at least initially, when the user is still in the process of learning to operate the prosthesis, but the effort is expected to decrease substantially with the training. Moving the prosthesis, interacting with the environment and contracting the muscles will likely change the perception of electrotactile stimulation. On one side, the active movements could distract the subject and thus negatively affect the interpretation of the feedback. However, it could also improve the SR in pattern discrimination since the user will know which patterns to expect, i.e., the ones that are congruent with the motor command he/ she sends to the prosthesis (e.g., open hand command will activate the aperture pattern). It could also minimize the confusion between the patterns representing the change of the DoF in the opposite directions (e.g. hand close versus hand open), because different muscles would be used to activate these movements. Furthermore, in the closed-loop application the user will be able to rely on the relative discrimination of levels, because the prosthesis states (e.g., force, aperture) will change continuously and gradually, rather than randomly, as was the case in the present psychometric evaluation.

Conclusion
This manuscript presents a novel system for electrotactile stimulation including fully programmable stimulator, custom designed flexible electrodes and a set of dynamic stimulation patterns communicating the state (proprioception and grasping force) of a multi-DoF prosthesis in an intuitive manner, i.e., spatial and intensity variables are mapped using spatial and intensity modulation. The tests demonstrated the lowlevel access to the system to implement spatial and frequency coding, and the set of high-level commands triggering dynamic patterns that were easy to adopt and identify by ablebodied subjects and amputees alike. In the present form, the system can accommodate four prosthesis DoFs, following the capabilities of the Michelangelo Hand, and can be used to provide absolute feedback regarding applied force, hand aperture and rotation through three continuous stimulation patterns, as well as relative feedback regarding hand flexion/ extension through the one discrete stimulation pattern. Moreover, the system can be connected to any prosthetic device equipped with a Bluetooth interface, and due to its programmability, the set of high-level commands and patterns can be easily adapted to different applications and coding schemes.