Introduction

Over the past decades, tremendous human and financial resources have been inputted into K-12 education. In order to understand whether such investment has improved the quality and efficiency of the education, we should first solve a core problem: how to quantitatively and qualitatively assess the teaching quality. Unfortunately, up to now, this problem still highly depends on feedbacks of domain experts, which is expensive and unscalable. In this paper, we propose an automatic teaching assistance system, called AIClass, by detecting and recognizing multiple characteristics during the class, including emotion states, speech analysis, action recognition, and position tracking. AIClass not only tracks and observes the teaching styles, but also comprehensively grasps the learning states of students. These structured analysis results could be mapped into the current curriculum observation schemes, and further realize the automatic and large-scale teaching assessment. It should be emphasized that the techniques used in AIClass are not the existing algorithms but creatively proposed to tackle the main challenges in real classroom environment. Experimental results demonstrate the effectiveness and efficiency of the proposed algorithms and AIClass system. In the future, we will further correlate features extracted from classrooms, and build a more intelligent teaching assistance system.

(1) Framework Overview

Summary of the four modules and their partial contents in AIClass system.

The system architecture of AIClass, which fully demonstrates the related components, vital details and processing flow, etc. Some logos are from official websites. Virtual icons are downloaded from the website https://www.flaticon.com/ freely.

(2) System Equipment

Left: Students captured by the frontend camera are having class in one ordinary classroom. In the red circle is the backend camera for the teacher. Faces are masked to protect privacy. Right: This picture is taken by the backend camera appearing in the left picture. The frontend camera is in the red circle. Center: We recommend installing hemispherical networked cameras with a resolution of 1K or 4K. In addition, some independent cameras need external pickup devices to collect voice.

(3) Algorithms and Results

The processing flow of all algorithms. We take our self-made datasets about audio and video of both teachers and students as inputs. Then, we will borrow off-the-shelf algorithms or rely on self-invented new methods to generate multiple solutions. The outputs are all basic features embedded in AIClass system.

An example of voice parsing using software Praat. Please focus on the pitch, intensity and syllable nuclei. Other more abundant phonetic features are auxiliary information when annotation.

Three examples of student behavior detection results. Yellow boxes are hand-raisings. The green, cyan and red boxes are standing, sleeping and yawning, respectively. Faces are masked to protect privacy.

The demonstration of some algorithm results related to students. It includes pose estimation, behavior matching and student location. The four point block diagrams on the right are flows of the student location algorithm. In the left, the final row and column numbers of students are drawn on their representative points in black with the format of RxCy. Faces are masked to protect privacy.

Head pose estimation qualitative results in two classrooms. Most of the faces or heads are well oriented. Faces enclosed by yellow solid circles are deserted difficult samples with low resolution or partial occlusion. Cubes with yellow dotted circles are all failures close to the backend.

Hard examples of hand-raising matching in real classroom. Skeletons are detected by the improved OpenPose. All poses have missing or false detected arm joints. The middle two samples are left hand-raisings. The right two samples are visually prone confusion. Faces are masked to protect privacy.

A series of qualitative detection and recognition examples related to the teacher algorithms. Bottom Right: One tracklet of teacher tracking and the teacher’s walking trajectory based on his or her position and projection transformation.

(3) Teaching Observation Methods

The matrix analysis of FIAS (Ned A Flanders. 1963. Intent, action and feedback: A preparation for teaching. Journal of Teacher Education).

The S-T analysis results of an example course. The subimage on the left is the S-T curve formed throughout the course. The right subimage is the Rt-Ch diagram with a red dot falling on the lecturing type area.

Declarations

The images introduced on this website regarding the AIClass project are not allowed to be reproduced or used without authorization, and violators will be held accountable.

The website template was borrowed from Jon Barron and Zip-NeRF.