Design Patterns and Frameworks for Developing WIMP+ User Interfaces

This work investigates the models and tools for support of developing a kind of future user interfaces, which are partially built upon the WIMP (Windows, Icons, Menus, and Pointing device: the mouse) interaction techniques and devices; and able to observe and leverage at least one controlled process under the supervision of their user(s). In this thesis, they are called WIMP+ user interfaces. There are a large variety of applications dealt with WIMP+ user interfaces, e.g., robot control, telecommunication, car driver assistant systems, distributed multi-user database systems, automation rail systems, etc. At first, it studies the evolution of user interfaces, deduces the innovative functions of future user interfaces, and defines WIMP+ user interfaces. Then, it investigates high level models for user interface realization. Since the most promising user-centered design methodology is a new emerging model, it is still short of modeling methodology and rules to support the concrete development process. Therefore, in this work, a universal modeling methodology, which picks up the design pattern application, is researched and used to structure different low level user interface models. And a framework, named Hot-UCDP, for aiding the development process, is proposed. Among the design patterns required by Hot-UCDP, this work puts its most effort on investigating user interface software architectural patterns. As a result, a WIMP+ user interface software architectural pattern Acquisition-Computation-Expression-Execution (ACEE) was discovered. Tools for user interface implementation are also surveyed. Based on the research results, an ACEE-based software framework prototype called Hot-WIMP+ was developed. To support programming WIMP+ user interfaces from multiple abstract levels, Hot-WIMP+ provides not only white and black box technique but also visual tools, e.g., Acquisition Definer, Expression Specification Tools, Execution Definer. The practicability of Hot-UCDP, ACEE, and Hot-WIMP+ is proven by applying them to create three WIMP+ user interfaces, LLDemo, Hot-WebRobi, and Hot-Demo for controlling a model robot. During their development, a robot application domain pattern SelectMe-ConveyMe-SettleMe and a human and computer interaction pattern Two-handed Manipulation were well researched and documented. SelectMe-ConveyMe-SettleMe could be, in principle, used as a starting point to develop other WIMP+ user interfaces for controlling the end-effector of a 3-D robot. Two-handed Manipulation could also be used to guide the design of bimanual interaction. The effect and efficiency of these WIMP+ user interfaces for support of users fulfilling telecontrol tasks are also investigated through a case study. Additionally, the combination of the Internet and telerobotics is also considered within this work.