Fix documentation for k3s/k8s usage of yaml files

Signed-off-by: Alexandre Peixoto Ferreira <alexandre.ferreira@arm.com>

.github/workflows/smarter-org-docker-buildx.yml

@@ -0,0 +1,21 @@
+name: Docker Image BuildX CI and Publish
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+on:
+  schedule:
+    - cron: "19 16 * * *"
+  push:
+    branches: ["main"]
+    # Publish semver tags as releases.
+    tags: ["v*.*.*"]
+  pull_request:
+    branches: ["main"]
+  workflow_dispatch:
+
+jobs:
+  build:
+    uses: smarter-project/reusable-workflows/.github/workflows/smarter-org-docker-buildx.yml@main

.github/workflows/smarter-org-helm.yml

@@ -0,0 +1,14 @@
+# release.yaml
+name: Release Charts
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  release:
+    uses: smarter-project/reusable-workflows/.github/workflows/smarter-org-helm.yml@main
+    secrets:
+      GPG_KEYRING_BASE64: ${{ secrets.GPG_KEYRING_BASE64 }}
+      GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}

.gitlab-ci.yml

@@ -1,6 +0,0 @@
-include:
-  - project: 'ericvh/gitlab-ci-arm-template'
-    file: '/.gitlab-ci.yml'
-
-variables:
-  CI_BUILDX_ARCHS: "linux/amd64,linux/arm64,linux/arm"

CONTRIBUTING.md

@@ -0,0 +1,16 @@
+# Contributing to SMARTER
+
+SMARTER is an open-source, software platform for the Edge. Contributions are an important part of the platform, and our goal is to make it as simple as possible to become a contributor.
+
+# Project license, and developer certificate of origin
+
+The SMARTER codebase is licensed under the Apache 2.0 open source license, individual components may have their own licenses. 
+Please see the LICENSE file in the SMARTER documentation repository for the full text of this license, or the LICENSE file
+in respective repositories.  New contributions are expected to be lisensed under the Apache 2.0 license or something compatible
+with that license.
+
+Please note that we expect contributors to the project to self-certify that they are authorized to contribute code 
+using the Linux Foundation's Developer Certificate of Origin. See http://developercertificate.org for more details.
+
+Contributors sign-off that they adhere to these requirements by adding a Signed-off-by line to commit messages.
+Pull requests for contributions without a signed-off-by line will not be accepted.

CONTRIBUTIONS

@@ -1,4 +0,0 @@
-We sincerely appreciate your interest of contributing to this project but for now
-we are not able to accept contributions. We expect to change this policy soon and
-provide a contribution procedure. We suggest that the project be maintained on a
-different branch until the contribution policy is changed.

Dockerfile

@@ -5,19 +5,18 @@ RUN apk update && apk upgrade && apk add tar ca-certificates build-base
 ENV GOPATH /go
 RUN go version
 
-WORKDIR /go/src/smarter-device-management
+WORKDIR /arm.com/smarter-device-management
 COPY . .
 
 RUN echo $PATH;export CGO_LDFLAGS_ALLOW='-Wl,--unresolved-symbols=ignore-in-object-files' && \
-    go install -ldflags="-s -w" -v smarter-device-management
+    go mod init arm.com/smarter-device-management && go mod tidy && go mod vendor && \
+    CGO_ENABLED=0 go build -ldflags='-s -w -extldflags="-static"' .
 
-FROM alpine
+FROM scratch
-
-RUN apk update && apk upgrade
 
 WORKDIR /root
 
 COPY conf.yaml /root/config/conf.yaml
-COPY --from=build /go/bin/smarter-device-management /usr/bin/smarter-device-management
+COPY --from=build /arm.com/smarter-device-management/smarter-device-management /usr/bin/smarter-device-management
 
-CMD ["smarter-device-management","-logtostderr=true","-v=0"]
+CMD ["/usr/bin/smarter-device-management","-logtostderr=true","-v=0"]

README.md

@@ -17,6 +17,10 @@ The smarter-device-manager starts by reading a YAML configuration file. This con
   nummaxdevices: 10
 ```
 
+Devices in subdirectories have the slash replaced with underscore in the
+resource name, due to kubernetes naming restrictions: e.g. `/dev/net/tun`
+becomes `smarter-devices/net_tun`.
+
 The default config file provided will enable most of the devices available on a Raspberry Pi (vers 1-4) or equivalent boards. I2C, SPI, video devices, sound and others would be enabled. The config file can be replaced using a configmap to enable or disable access to different devices, like accelerators, GPUs, etc.
 
 The node will show the devices it recognizes as resources in the node object in Kubernetes. The example below shows a raspberry PI.
@@ -133,7 +137,7 @@ More than one smarter-device-manager can be used in a single node if required if
 
 ## Enabling Access
 
-A few examples of yaml files are provided that enable the smarter-device-manager to be deployed in a node. The file smarter-device-management-pod-<>.yaml deploys a single pod on a node; this setup is useful for testing. The file smarter-device-manager-<>.yaml provides a deamonSet configuration that enables pods to be deployed in any node that contains the "smarter-device-manager=enabled" label. The following command inserts the daemonSet in Kubernetes. k3s and k8s put the unix sockets for the device plugin in different directories on the node so the \*-k8s.yaml files should be used on Kubernetes and the \*-k3s.yaml should be used on k3s.
+A few examples of yaml files are provided that enable the smarter-device-manager to be deployed in a node. The file smarter-device-management-pod-<>.yaml deploys a single pod on a node; this setup is useful for testing. The file smarter-device-manager-<>.yaml provides a deamonSet configuration that enables pods to be deployed in any node that contains the "smarter-device-manager=enabled" label. The following command inserts the daemonSet in Kubernetes. Use the k8s for k8s/k3s/k0s unless using k3s version lower than 1.18. K3s smaller then 1.18 put the unix sockets for the device plugin in different directories on the node so the \*-k3s.yaml files should be used on k3s for those versions.
 
 ```
 kubectl apply -f smarter-device-manager.yaml
@@ -147,3 +151,13 @@ The following command should show the node resources in a similar form as shown
 kubectl describe node pike5
 ```
 
+## k3s 
+
+K3s < 1.18 stores the plugin interface in a different directory than k8s and so it needs a different yaml file to enable smarter-device-manager to communicate correctly with k3s agent. So use the smart-device-manager-k3s yaml files on this reposistor for k3s < 1.18.
+
+## Using helm
+
+A helm chart that install smarter-device-manager configured for SMARTER is available at chart directory
+```
+helm install smarter-device-manager chart
+```

argus.go

@@ -4,7 +4,7 @@ package main
 
 import (
 	"github.com/golang/glog"
-	pluginapi "k8s.io/kubernetes/pkg/kubelet/apis/deviceplugin/v1beta1"
+        pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1"
 )
 
 func check(err error) {

charts/smarter-device-manager/.helmignore

@@ -0,0 +1,26 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/
+# OWNERS file for Kubernetes
+OWNERS
+# helm-docs templates
+*.gotmpl

charts/smarter-device-manager/Chart.yaml

@@ -0,0 +1,26 @@
+apiVersion: v2
+name: smarter-device-manager
+description: smarter-device-manager chart for SMARTER edge devices
+home: https://getsmarter.io
+version: 0.0.9
+appVersion: v1.20.12
+kubeVersion: ">=1.16.0-0"
+keywords:
+  - kubernetes
+  - device
+  - hardware
+sources:
+  - https://github.com/smarter-project/smarter-device-manager
+icon: https://github.com/smarter-project/documentation/raw/main/ARM1636_Project_Logo_ST2_RGB_V1.png
+annotations:
+  artifacthub.io/changes: |
+    - Fix template
+    - Add annotations
+  artifacthub.io/license: Apache-2.0
+  artifacthub.io/maintainers: |
+    - name: Alexandre Ferreira
+      email: alexandref75@gmail.com
+  artifacthub.io/prerelease: "false"
+  artifacthub.io/signKey: |
+    fingerprint: 71EDA4E3D652DC73EB09E3A5387D298C169CF24E
+    url: https://smarter-project.github.io/documentation/pgp_keys.asc

charts/smarter-device-manager/README.md

@@ -0,0 +1,57 @@
+# SMARTER Device Manager
+
+Enables k8s containers to access devices (linux device drivers) available on nodes.
+
+For more information check out https://getsmarter.io
+
+## TL;DR
+
+Assumes that this repository was cloned.
+
+```console
+helm install --nsmespace=smarter --create-namespace my-smarter-device-manager charts/smarter-device-manager
+```
+
+## Overview
+
+In the IoT world, interaction with the external environment is the reason of existence.
+This interaction is done by acquiring data about the environment and, possibly, actuating to achieve the desired objective, with complexity ranging from a simple thermostat to a very complex industrial process control (e.g. chemical plant). In more practical terms, the main CPU interacts directly with those sensors and actuators and the OS (Linux in our case) provides an abstract view in the form of device drivers.
+Even though the container runtime allows direct access to device drivers, containers running on Kubernetes in the cloud are not expected to do so since hardware independence is a very useful characteristic to enhance mobility.
+Kubernetes primarily manages CPU, memory, storage, and network, while leaving other resources unmanaged.
+In IoT environments, applications can have direct access to sensors and actuators, either directly by interfacing with a device driver on the kernel (e.g. digital I/O pins, temperature sensors, analog inputs, microphones, audio output, video cameras) or indirectly through hardware interfaces (like serial ports, I2C, SPI, bluetooth, LoRa, USB and others).
+Controlled access to these devices is essential to enable a container-based IoT solution. Smarter-device-manager allows containers to have direct access to host devices in a secure way.
+
+## Values
+
+The configuration.nodeSelector value allows the nodeSelector to be changed in a higher level chart simplyfyng deploying multiple services at the same time; CNI, DNS and device-manager with a single label for example.
+
+## Pre-requisites
+
+- k8s > 1.18 (before this the plugin interface used a different directory which requires a different configuration)
+- by default, smarter-device manager uses a node-select to choose which nodes to deploy to, so label your nodes appropriately in order to deploy:
+```
+kubectl label node mynode01 smarter-device-manager=enabled
+```
+
+## Usage Model
+
+The smarter-device-manager starts by reading a YAML configuration file. This configuration file describes, using regular expressions, the files that identify each device that is to be exported and how many access can be done simultaneously. For example, the configuration below finds every V4L device (cameras, video tuners, etc...) available on the host node (/dev/video0, /dev/video1, etc), and adds them as resources (smarter-devices/video0, smarter-devices/video1, etc) that allow up to 10 simulatenous accesses (up to 10 containers can request access to those devices simultaneously). 
+```
+- devicematch: ^video[0-9]*$
+  nummaxdevices: 10
+```
+
+If the config value is provided a configMap is generated and smarter-device-manager will use it. The values.yaml file contains two examples, the first is replicated the config that exists on the container and the second enables nitro-enclaves (AWS nitro).
+
+Devices in subdirectories have the slash replaced with underscore in the
+resource name, due to kubernetes naming restrictions: e.g. `/dev/net/tun`
+becomes `smarter-devices/net_tun`.
+
+The default config file provided will enable most of the devices available on a Raspberry Pi (vers 1-4) or equivalent boards. I2C, SPI, video devices, sound and others would be enabled. The config file can be replaced using a configmap to enable or disable access to different devices, like accelerators, GPUs, etc.
+
+# Uninstalling the Chart
+
+```
+helm delete my-smarter-device-manager
+
+```

charts/smarter-device-manager/templates/common.yaml

@@ -1,50 +1,49 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: < Replace with the namespace to use >
-  labels:
-    name: < Replace with the namespace to use >
----
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
-  name: smarter-device-manager
+  name: {{ .Values.application.appName }}
-  namespace: < Replace with the namespace to use >
   labels:
-    name: smarter-device-manager
+    name: {{ .Values.application.appName }}
     role: agent
 spec:
   selector:
     matchLabels:
-      name: smarter-device-manager
+      name: {{ .Values.application.appName }}
   updateStrategy:
     type: RollingUpdate
   template:
     metadata:
       labels: 
-        name: smarter-device-manager
+        name: {{ .Values.application.appName }}
       annotations:
         node.kubernetes.io/bootstrap-checkpoint: "true"
     spec: 
       nodeSelector:
-        smarter-device-manager : enabled
+        {{- if .Values.nodeSelector }}
+        {{- toYaml .Values.nodeSelector | nindent 8 }}
+        {{- else }}
+        smarter.device-manager: enabled
+        {{- end }}
+      tolerations:
+      - key: "smarter.type"
+        operator: "Equal"
+        value: "edge"
+        effect: "NoSchedule"
       priorityClassName: "system-node-critical"
-      hostname: smarter-device-management
+      hostname: {{ .Values.application.appName }}
       hostNetwork: true
       dnsPolicy: ClusterFirstWithHostNet
-      imagePullSecrets:
-      - name: k8sedgeregcred
       containers:
-      - name: smarter-device-manager
+      - name: {{ .Values.application.appName }}
-        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+        image: {{ .Values.image.repository }}:{{ default .Chart.AppVersion .Values.image.tag }}
-        imagePullPolicy: IfNotPresent
+        imagePullPolicy: {{ .Values.image.pullPolicy }}
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
             drop: ["ALL"]
         resources:
           limits:
-            cpu: 100m
+            cpu: 200m
             memory: 15Mi
           requests:
             cpu: 10m
@@ -55,23 +54,24 @@ spec:
           - name: dev-dir
             mountPath: /dev
           - name: sys-dir
-             mountPath: /sys
+            mountPath: /sys
+          {{- if .Values.config }}
           - name: config
             mountPath: /root/config
+          {{- end }}
       volumes:
         - name: device-plugin
           hostPath:
-            path: /var/lib/rancher/k3s/agent/kubelet/device-plugins
+            path: /var/lib/kubelet/device-plugins
         - name: dev-dir
           hostPath:
             path: /dev
         - name: sys-dir
           hostPath:
             path: /sys
+        {{- if .Values.config }}
         - name: config
           configMap:
-             name: smarter-device-manager-rpi
+            name: {{ .Values.application.appName }}
-        - name: config
+        {{- end }}
-          hostPath:
-            path: /var/lib/rancher/k3s/agent/kubelet/device-plugins
       terminationGracePeriodSeconds: 30

charts/smarter-device-manager/templates/configmap.yaml

@@ -0,0 +1,9 @@
+{{- if .Values.config }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Values.application.appName }}
+data:
+  conf.yaml: |
+  {{- toYaml .Values.config | nindent 4 }}
+{{- end }}

charts/smarter-device-manager/values.yaml

@@ -0,0 +1,57 @@
+#
+
+application:
+  appName: smarter-device-manager
+
+image:
+  repository: ghcr.io/smarter-project/smarter-device-manager
+  # @default -- chart.appVersion
+  tag: ""
+  pullPolicy: IfNotPresent
+
+# If a specific configurations is used it can be provided by uncommenting this lines
+# config:
+#   - devicematch: ^snd$
+#     nummaxdevices: 20
+#   - devicematch: ^gpiomem$
+#     nummaxdevices: 40
+#   - devicematch: ^gpiochip[0-9]*$
+#     nummaxdevices: 20
+#   - devicematch: ^hci[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^i2c-[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^rtc0$
+#     nummaxdevices: 20
+#   - devicematch: ^video[0-9]*$
+#     nummaxdevices: 20
+#   - devicematch: ^vchiq$
+#     nummaxdevices: 20
+#   - devicematch: ^vcsm.*$
+#     nummaxdevices: 20
+#   - devicematch: ^ttyUSB[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^ttyACM[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^ttyTHS[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^ttyS[0-9]*$
+#     nummaxdevices: 1
+#
+
+# This example of configmap allows nitro enclaves to be allocated in a pod
+# config:
+#   - devicematch: ^nitro_enclaves$
+#     nummaxdevices: 1
+#   - devicematch: ^vsock$
+#     nummaxdevices: 1
+#   - devicematch: ^rtc0$
+#     nummaxdevices: 20
+#   - devicematch: ^ttyUSB[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^ttyACM[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^ttyTHS[0-9]*$
+#     nummaxdevices: 1
+#   - devicematch: ^ttyS[0-9]*$
+#     nummaxdevices: 1

compile.sh

@@ -5,6 +5,12 @@
 function printHelp() {
                 echo $(basename $0)" options:";
                 echo "    -A <Architectures to use> # Compiling to ${ARCHS} now, examples: linux/amd64,linux/arm/v7,linux/arm/v6,linux/arm64"
+                if [ ${FLAG_NOCACHE} -gt 0 ]
+                then
+                        echo "    -C # Do not use cache"
+                else 
+                        echo "    -C # Use cache" 
+                fi
                 if [ ${FLAG_UPLOADIMAGES} -gt 0 ]
                 then
                         echo "    -U # Do not upload images - the default is upload the images to the registry"
@@ -30,7 +36,7 @@ function printHelp() {
 }
 
 BUILD_TAG=$(date -u "+%Y%m%d%H%M%S")
-REPOSITORY_NAME="registry.gitlab.com/arm-research/smarter/smarter-device-manager/"
+REPOSITORY_NAME="ghcr.io/smarter-project/smarter-device-manager/"
 IMAGE_NAME="smarter-device-manager"
 DIRECTORY_TO_RUN=.
 
@@ -43,8 +49,9 @@ FLAG_UPLOADMANIFEST=1
 ADDITIONAL_TAG=""
 ADDITIONAL_IMAGE_NAME=""
 PUSH_OPTION=""
+FLAG_NOCACHE=0
 
-while getopts hA:B:MST:U name
+while getopts hA:B:MST:UC name
 do
         case $name in
         h)
@@ -52,6 +59,10 @@ do
                 exit 0;;
         A)
                 ARCHS="$OPTARG";;
+        C)
+                [ ${FLAG_NOCACHE} -gt 0 ] && FLAG_NOCACHE=0;
+                [ ${FLAG_NOCACHE} -eq 0 ] && FLAG_NOCACHE=1;
+                ;;
         U)
                 [ ${FLAG_UPLOADIMAGES} -gt 0 ] && FLAG_UPLOADIMAGES=0;
                 [ ${FLAG_UPLOADIMAGES} -eq 0 ] && FLAG_UPLOADIMAGES=1;
@@ -93,6 +104,13 @@ EOF
         fi
 fi
 
+if [ $FLAG_NOCACHE -gt 0 ]
+then
+        CACHE_OPTION="--no-cache"
+else
+        CACHE_OPTION="" 
+fi
+        
 if [ $FLAG_UPLOADIMAGES -gt 0 ]
 then
         PUSH_OPTION="--push"
@@ -100,6 +118,6 @@ else
         PUSH_OPTION="--load"
 fi
         
-docker buildx build  -t "${REPOSITORY_NAME}${IMAGE_NAME}${ADDITIONAL_IMAGE_NAME}:${BUILD_TAG}" --platform=${ARCHS} ${PUSH_OPTION} .
+docker buildx build ${CACHE_OPTION}  -t "${REPOSITORY_NAME}${IMAGE_NAME}${ADDITIONAL_IMAGE_NAME}:${BUILD_TAG}" --platform=${ARCHS} ${PUSH_OPTION} .
 
 exit 0

main.go

@@ -14,7 +14,7 @@ import (
 	"github.com/fsnotify/fsnotify"
 	"github.com/golang/glog"
 	"gopkg.in/yaml.v2"
-	pluginapi "k8s.io/kubernetes/pkg/kubelet/apis/deviceplugin/v1beta1"
+        pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1"
 )
 
 var confFileName string
@@ -55,18 +55,60 @@ func init() {
 	flag.Parse()
 }
 
-func readDevDirectory(dirToList string) (files []string, err error) {
+func readDevDirectory(dirToList string, allowedRecursions uint8) (files []string, err error) {
-	f, err := os.Open(dirToList)
+        var foundFiles []string
-	if err != nil {
-		return nil, err
-	}
-	files, err = f.Readdirnames(-1)
-	f.Close()
-	if err != nil {
-		return nil, err
-	}
 
-	return files, nil
+        fType, err := os.Stat(dirToList)
+        if err != nil {
+                return nil, err
+        }
+
+        if !fType.IsDir()  {
+                return nil, nil
+        }
+
+        f, err := os.Open(dirToList)
+        if err != nil {
+                return nil, err
+        }
+        files, err = f.Readdirnames(-1)
+        if err != nil {
+                f.Close()
+                return nil, err
+        }
+        f.Close()
+        for _, subDir := range files {
+                foundFiles = append(foundFiles, subDir)
+                if allowedRecursions > 0 {
+                        filesDir, err := readDevDirectory(dirToList+"/"+subDir,allowedRecursions-1)
+                        if err == nil {
+                                for _, fileName := range filesDir {
+                                        foundFiles = append(foundFiles, subDir+"/"+fileName)
+                                }
+                        }
+                }
+        }
+
+        return foundFiles, nil
+}
+
+func sanitizeName(path string) string {
+        sanitizeChar := func(r rune) rune {
+		switch {
+		case r >= 'A' && r <= 'Z':
+			return r
+		case r >= 'a' && r <= 'z':
+			return r
+		case r >= '0' && r <= '9':
+			return r
+		case r == '_':
+			return r
+		case r == '-':
+			return r
+		}
+		return '_'
+	}
+        return strings.Map(sanitizeChar, path)
 }
 
 func findDevicesPattern(listDevices []string, pattern string) ([]string,error) {
@@ -102,13 +144,13 @@ func main() {
         }
 
 	glog.V(0).Info("Reading existing devices on /dev")
-	ExistingDevices, err := readDevDirectory("/dev")
+	ExistingDevices, err := readDevDirectory("/dev",10)
 	if err != nil {
 		glog.Errorf(err.Error())
 		os.Exit(1)
 	}
 
-	ExistingDevicesSys, err := readDevDirectory("/sys/devices")
+	ExistingDevicesSys, err := readDevDirectory("/sys/devices",0)
 	if err != nil {
 		glog.Errorf(err.Error())
 		os.Exit(1)
@@ -151,9 +193,10 @@ func main() {
                         if len(foundDevices) > 0 {
                                 for _, deviceToCreate := range foundDevices {
                                         var newDevice DeviceInstance
+                                        deviceSafeName := sanitizeName(deviceToCreate)
                                         newDevice.deviceType = deviceFileType
-                                        newDevice.deviceName = "smarter-devices/" + deviceToCreate
+                                        newDevice.deviceName = "smarter-devices/" + deviceSafeName
-                                        newDevice.socketName = pluginapi.DevicePluginPath + "smarter-" + deviceToCreate + ".sock"
+                                        newDevice.socketName = pluginapi.DevicePluginPath + "smarter-" + deviceSafeName + ".sock"
                                         newDevice.deviceFile = "/dev/" + deviceToCreate
                                         newDevice.numDevices = deviceToTest.NumMaxDevices
                                         listDevicesAvailable = append(listDevicesAvailable, newDevice)
@@ -193,17 +236,17 @@ L:
 			}
 
 			var err error
-			for _, devicesInUse := range listDevicesAvailable {
+			for id, _ := range listDevicesAvailable {
-                                switch devicesInUse.deviceType {
+                                switch listDevicesAvailable[id].deviceType {
                                 case deviceFileType :
-                                        devicesInUse.devicePluginSmarter = NewSmarterDevicePlugin(devicesInUse.numDevices, devicesInUse.deviceFile, devicesInUse.deviceName, devicesInUse.socketName)
+                                        listDevicesAvailable[id].devicePluginSmarter = NewSmarterDevicePlugin(listDevicesAvailable[id].numDevices, listDevicesAvailable[id].deviceFile, listDevicesAvailable[id].deviceName, listDevicesAvailable[id].socketName)
-                                        if err = devicesInUse.devicePluginSmarter.Serve(); err != nil {
+                                        if err = listDevicesAvailable[id].devicePluginSmarter.Serve(); err != nil {
                                                 glog.V(0).Info("Could not contact Kubelet, retrying. Did you enable the device plugin feature gate?")
                                                 break
                                         }
                                 case nvidiaSysType :
-                                        devicesInUse.devicePluginNvidia = NewNvidiaDevicePlugin(devicesInUse.numDevices, devicesInUse.deviceName,"NVIDIA_VISIBLE_DEVICES", devicesInUse.socketName, devicesInUse.deviceId)
+                                        listDevicesAvailable[id].devicePluginNvidia = NewNvidiaDevicePlugin(listDevicesAvailable[id].numDevices, listDevicesAvailable[id].deviceName,"NVIDIA_VISIBLE_DEVICES", listDevicesAvailable[id].socketName, listDevicesAvailable[id].deviceId)
-                                        if err = devicesInUse.devicePluginNvidia.Serve(); err != nil {
+                                        if err = listDevicesAvailable[id].devicePluginNvidia.Serve(); err != nil {
                                                 glog.V(0).Info("Could not contact Kubelet, retrying. Did you enable the device plugin feature gate?")
                                                 break
                                         }
@@ -234,13 +277,18 @@ L:
 			default:
 				glog.V(0).Infof("Received signal \"%v\", shutting down.", s)
 				for _, devicesInUse := range listDevicesAvailable {
+                                        glog.V(0).Info("Stopping device ", devicesInUse.deviceName)
                                         switch devicesInUse.deviceType {
                                         case deviceFileType :
+				                glog.V(0).Info("Smarter device type")
                                                 if devicesInUse.devicePluginSmarter != nil {
+				                        glog.V(0).Info("Stopping device")
                                                         devicesInUse.devicePluginSmarter.Stop()
                                                 }
                                         case nvidiaSysType :
+				                glog.V(0).Info("Nvidia device type")
                                                 if devicesInUse.devicePluginNvidia != nil {
+				                        glog.V(0).Info("Stopping device")
                                                         devicesInUse.devicePluginNvidia.Stop()
                                                 }
                                         }

nvidia-server.go

@@ -13,7 +13,7 @@ import (
 	"github.com/golang/glog"
 	"golang.org/x/net/context"
 	"google.golang.org/grpc"
-	pluginapi "k8s.io/kubernetes/pkg/kubelet/apis/deviceplugin/v1beta1"
+        pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1"
 )
 
 var passDeviceSpecs = flag.Bool("pass-device-specs", false, "pass the list of DeviceSpecs to the kubelet on Allocate()")
@@ -182,6 +182,10 @@ func (m *NvidiaDevicePlugin) PreStartContainer(context.Context, *pluginapi.PreSt
 	return &pluginapi.PreStartContainerResponse{}, nil
 }
 
+func (m *NvidiaDevicePlugin) GetPreferredAllocation(context.Context, *pluginapi.PreferredAllocationRequest) (*pluginapi.PreferredAllocationResponse, error) {
+	return &pluginapi.PreferredAllocationResponse{}, nil
+}
+
 func (m *NvidiaDevicePlugin) cleanup() error {
 	if err := os.Remove(m.socket); err != nil && !os.IsNotExist(err) {
 		return err

server.go

@@ -13,7 +13,7 @@ import (
 	"github.com/golang/glog"
 	"golang.org/x/net/context"
 	"google.golang.org/grpc"
-	pluginapi "k8s.io/kubernetes/pkg/kubelet/apis/deviceplugin/v1beta1"
+        pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1"
 )
 
 const (
@@ -94,6 +94,7 @@ func (m *SmarterDevicePlugin) Start() error {
 
 // Stop the gRPC server
 func (m *SmarterDevicePlugin) Stop() error {
+	glog.V(0).Infof("Stopping server with socket ",m.socket)
 	if m.server == nil {
 		return nil
 	}
@@ -101,6 +102,7 @@ func (m *SmarterDevicePlugin) Stop() error {
 	m.server.Stop()
 	m.server = nil
 	close(m.stop)
+	glog.V(0).Info("Server stopped with socket ",m.socket)
 
 	return m.cleanup()
 }
@@ -178,7 +180,12 @@ func (m *SmarterDevicePlugin) PreStartContainer(context.Context, *pluginapi.PreS
 	return &pluginapi.PreStartContainerResponse{}, nil
 }
 
+func (m *SmarterDevicePlugin) GetPreferredAllocation(context.Context, *pluginapi.PreferredAllocationRequest) (*pluginapi.PreferredAllocationResponse, error) {
+	return &pluginapi.PreferredAllocationResponse{}, nil
+}
+
 func (m *SmarterDevicePlugin) cleanup() error {
+	glog.V(0).Info("Removing file ",m.socket)
 	if err := os.Remove(m.socket); err != nil && !os.IsNotExist(err) {
 		return err
 	}

smarter-device-management-pod-k3s-test-xavier.yaml

@@ -15,7 +15,7 @@ spec:
   nodeName: smarter-jetson-xavier-4bcc2584
   containers:
   - name: smarter-device-manager
-    image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+    image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
     imagePullPolicy: IfNotPresent
     securityContext:
       allowPrivilegeEscalation: false

smarter-device-management-pod-k3s.yaml

@@ -15,7 +15,7 @@ spec:
   nodeName: <replace with node to run>
   containers:
   - name: smarter-device-manager
-    image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+    image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
     imagePullPolicy: IfNotPresent
     securityContext:
       allowPrivilegeEscalation: false

smarter-device-management-pod-k8s.yaml

@@ -1,43 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: smarter-device-management
-  namespace: default
-spec:
-  # Mark this pod as a critical add-on; when enabled, the critical add-on
-  # scheduler reserves resources for critical add-on pods so that they can
-  # be rescheduled after a failure.
-  # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
-  priorityClassName: "system-node-critical"
-  hostNetwork: true
-  dnsPolicy: ClusterFirstWithHostNet
-  hostname: smarter-device-management
-  nodeName: <replace with node to run>
-  containers:
-  - name: smarter-device-manager
-    image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
-    imagePullPolicy: IfNotPresent
-    securityContext:
-      allowPrivilegeEscalation: false
-      capabilities:
-        drop: ["ALL"]
-    resources:
-      limits:
-        cpu: 100m
-        memory: 10Mi
-      requests:
-        cpu: 10m
-        memory: 10Mi
-    volumeMounts:
-      - name: device-plugin
-        mountPath: /var/lib/kubelet/device-plugins
-      - name: dev-dir
-        mountPath: /dev
-  volumes:
-    - name: device-plugin
-      hostPath:
-        path: /var/lib/kubelet/device-plugins
-    - name: dev-dir
-      hostPath:
-            path: /dev
-      terminationGracePeriodSeconds: 30

smarter-device-management-pod.yaml

@@ -12,10 +12,10 @@ spec:
   hostNetwork: true
   dnsPolicy: ClusterFirstWithHostNet
   hostname: smarter-device-management
-  nodeName: smarter-jetson-xavier-4bcc2584
+  nodeName: <replace with node to run>
   containers:
   - name: smarter-device-manager
-    image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+    image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
     imagePullPolicy: IfNotPresent
     securityContext:
       allowPrivilegeEscalation: false
@@ -38,7 +38,7 @@ spec:
   volumes:
     - name: device-plugin
       hostPath:
-        path: /var/lib/rancher/k3s/agent/kubelet/device-plugins
+        path: /var/lib/kubelet/device-plugins
     - name: dev-dir
       hostPath:
             path: /dev

smarter-device-manager-configmap-rpi.yaml

@@ -22,3 +22,11 @@ data:
           nummaxdevices: 20
         - devicematch: ^vcsm.*$
           nummaxdevices: 20
+        - devicematch: ^ttyUSB[0-9]*$
+          nummaxdevices: 1
+        - devicematch: ^ttyACM[0-9]*$
+          nummaxdevices: 1
+        - devicematch: ^ttyTHS[0-9]*$
+          nummaxdevices: 1
+        - devicematch: ^ttyS[0-9]*$
+          nummaxdevices: 1

smarter-device-manager-configmap-xavier.yaml

@@ -22,5 +22,13 @@ data:
           nummaxdevices: 20
         - devicematch: ^vcsm.*$
           nummaxdevices: 20
+        - devicematch: ^ttyUSB[0-9]*$
+          nummaxdevices: 1
+        - devicematch: ^ttyACM[0-9]*$
+          nummaxdevices: 1
+        - devicematch: ^ttyTHS[0-9]*$
+          nummaxdevices: 1
+        - devicematch: ^ttyS[0-9]*$
+          nummaxdevices: 1
         - devicematch: nvidia-gpu
           nummaxdevices: 20

smarter-device-manager-ds-k3s.yaml

@@ -34,7 +34,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       containers:
       - name: smarter-device-manager
-        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+        image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
         imagePullPolicy: IfNotPresent
         securityContext:
           allowPrivilegeEscalation: false

smarter-device-manager-ds-with-configmap-rpi-k3s.yaml

@@ -0,0 +1,72 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: < Replace with the namespace to use >
+  labels:
+    name: < Replace with the namespace to use >
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smarter-device-manager
+  namespace: < Replace with the namespace to use >
+  labels:
+    name: smarter-device-manager
+    role: agent
+spec:
+  selector:
+    matchLabels:
+      name: smarter-device-manager
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels: 
+        name: smarter-device-manager
+      annotations:
+        node.kubernetes.io/bootstrap-checkpoint: "true"
+    spec: 
+      nodeSelector:
+        smarter-device-manager : enabled
+      priorityClassName: "system-node-critical"
+      hostname: smarter-device-management
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
+      containers:
+      - name: smarter-device-manager
+        image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
+        imagePullPolicy: IfNotPresent
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop: ["ALL"]
+        resources:
+          limits:
+            cpu: 100m
+            memory: 15Mi
+          requests:
+            cpu: 10m
+            memory: 15Mi
+        volumeMounts:
+          - name: device-plugin
+            mountPath: /var/lib/kubelet/device-plugins
+          - name: dev-dir
+            mountPath: /dev
+          - name: config
+            mountPath: /root/config
+          - name: sys-dir
+            mountPath: /sys
+      volumes:
+        - name: device-plugin
+          hostPath:
+            path: /var/lib/rancher/k3s/agent/kubelet/device-plugins
+        - name: dev-dir
+          hostPath:
+            path: /dev
+        - name: sys-dir
+          hostPath:
+            path: /sys
+        - name: config
+          configMap:
+             name: smarter-device-manager-rpi
+      terminationGracePeriodSeconds: 30

smarter-device-manager-ds-with-configmap-rpi.yaml

@@ -32,11 +32,9 @@ spec:
       hostname: smarter-device-management
       hostNetwork: true
       dnsPolicy: ClusterFirstWithHostNet
-      imagePullSecrets:
-      - name: k8sedgeregcred
       containers:
       - name: smarter-device-manager
-        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+        image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
         imagePullPolicy: IfNotPresent
         securityContext:
           allowPrivilegeEscalation: false

smarter-device-manager-ds-with-configmap-xavier-k3s.yaml

@@ -34,7 +34,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       containers:
       - name: smarter-device-manager
-        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+        image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
         imagePullPolicy: IfNotPresent
         securityContext:
           allowPrivilegeEscalation: false
@@ -52,10 +52,10 @@ spec:
             mountPath: /var/lib/kubelet/device-plugins
           - name: dev-dir
             mountPath: /dev
+          - name: sys-dir
+             mountPath: /sys
           - name: config
             mountPath: /root/config
-          - name: sys-dir
-            mountPath: /sys
       volumes:
         - name: device-plugin
           hostPath:

smarter-device-manager-ds-with-configmap-xavier.yaml

@@ -32,11 +32,9 @@ spec:
       hostname: smarter-device-management
       hostNetwork: true
       dnsPolicy: ClusterFirstWithHostNet
-      imagePullSecrets:
-      - name: k8sedgeregcred
       containers:
       - name: smarter-device-manager
-        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+        image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
         imagePullPolicy: IfNotPresent
         securityContext:
           allowPrivilegeEscalation: false

smarter-device-manager-ds.yaml

@@ -32,11 +32,9 @@ spec:
       hostname: smarter-device-management
       hostNetwork: true
       dnsPolicy: ClusterFirstWithHostNet
-      imagePullSecrets:
-      - name: k8sedgeregcred
       containers:
       - name: smarter-device-manager
-        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:IMAGE_ID
+        image: ghcr.io/smarter-project/smarter-device-manager:v1.20.11
         imagePullPolicy: IfNotPresent
         securityContext:
           allowPrivilegeEscalation: false